sig: the sequel

Did Obama’s federal school turnaround program really fail?

Education Secretary Arne Duncan and HHS Secretary Kathleen Sebelius read to students enrolled in a Head Start program at Rolling Terrace Elementary School March 1, 2013 in Takoma Park, Maryland. (Photo by Win McNamee/Getty Images)

It has become a talking point for Betsy DeVos and a powerful example of the challenges of turning around struggling schools: a national study, released by the federal government, showing that its multibillion-dollar turnaround program failed.

“The previous administration spent seven billion of your dollars on ‘School Improvement Grants,’ thinking they could demonstrate that money alone would solve the problem,” DeVos said soon after becoming education secretary. “They tested their model, and it failed miserably.”

A new report says, not so fast. It points to studies of places like San Francisco, where the approach seemed to help students, and the limitations of the government’s study to conclude that the federal report painted too grim a picture.

“The autopsy on the grant program is flawed and its core conclusion faulty,” says the analysis, released by FutureEd, a Georgetown-based think tank generally supportive of Obama-era education policies.

It’s a year-old debate that remains relevant, as the study has become a touchstone for the idea that the federal government is unable to help long-struggling schools improve. And it comes as states, now with more freedom, are grappling with how to intervene in their lowest performing schools.

The federal turnaround program, known as School Improvement Grants or SIG, was a signature initiative of the Obama administration. In exchange for federal money, schools had to make changes, but had to use one of four approaches. About three-quarters chose the least disruptive option: firing the principal and making adjustments like lengthening the school day or toughening their teacher evaluations. Others replaced the principal and half the teaching staff. Few chose the other options, turning a school over to charter school operator or closing it altogether.

The initiative was evaluated by the external research firm Mathematica and the American Institutes for Research and released by the education department’s research arm, the Institute of Education Sciences. The results, released in January 2017, weren’t pretty.

“There was also no evidence that SIG had significant impacts on math or reading test scores, high school graduation, or college enrollment,” the study said.

The FutureEd report, written by two former Department of Education officials, suggests those conclusions are flawed for a few reasons, some of which were noted when the study was first released.

For one, it points out that even if School Improvement Grants successfully improved students’ academic outcomes, that would have been hard for the study to detect because the study’s bar for “statistical significance” was a very high one to clear.

The study actually estimated that the grants led to modest boosts in reading test scores and small declines in high school graduation rates — but neither impact was statistically significant, which is what the researchers mean when they say they found “no evidence.”

One of the report’s researchers, Lisa Dragoset of Mathematica, defended its approach. Setting a high bar for significance is reasonable for a program, like SIG, that was quite costly, she said. And even ignoring statistical significance, the estimated gains in reading were small and essentially zero in math, she noted.

“It’s unlikely that there were substantive or large impacts that were undetected by our study,” she said.

Second, FutureEd points out that the subset of schools in the federal study were not representative of all schools receiving federal grants.

The study compared schools receiving SIG grants to those schools near the eligibility cutoff that didn’t get grants. This is a widely used approach, but the FutureEd authors point out that the results then don’t say much about the effects of the grants on the lowest-performing schools. The studied schools were also disproportionately urban.

The new report also highlights a number of studies that focus on specific states and cities which paint a much more positive picture of the initiative. Most, though not all, of these studies find that the grants had positive effects on test scores.

“There are legitimate questions of whether the SIG program represented the best way to use federal funding to improve struggling schools,” the FutureEd authors conclude. “But it is wrong to suggest that there was no return on the SIG investment.”

Dragoset acknowledges that national results might not apply to all SIG schools, but says there’s nothing inconsistent with seeing no clear national effect but positive results in certain states.

“Ours, to my knowledge, is the only large-scale, rigorous study of the SIG program nationwide,” she said.

Dan Goldhaber, a University of Washington professor and vice president at the American Institutes for Research who reviewed the FutureEd report, said the limits of the federal study suggest policymakers shouldn’t “jump to the conclusion that the SIG program didn’t work.” But the study “was competently done,” he said.

The FutureEd analysis concludes that the U.S. Department of Education should spearhead a review of the research on turning around struggling schools. Tom Dee, a Stanford professor who found that the federal grants led to higher test scores in California schools, echoed this.

“I believe the question we should be asking is the following: why do federal reform catalysts seem to generate positive change in some states and communities but mere cosmetic regulatory compliance in others?” he said. “It seems to me that knowing more about the answer to that question is critical to efforts to drive meaningful change at scale.”

Mapping a Turnaround

This is what the State Board of Education hopes to order Adams 14 to do

PHOTO: Hyoung Chang/The Denver Post
Javier Abrego, superintendent of Adams 14 School District on April 17, 2018.

In Colorado’s first-ever attempt to give away management of a school district, state officials Thursday provided a preview of what the final order requiring Adams 14 to give up district management could include.

The State Board of Education is expected to approve its final directives to the district later this month.

Thursday, after expressing a lack of trust in district officials who pleaded their case, the state board asked the Attorney General’s office for advice and help in drafting a final order detailing how the district is to cede authority, and in what areas.

Colorado has never ordered an external organization to take over full management of an entire district.

Among details discussed Thursday, Adams 14 will be required to hire an external manager for at least four years. The district will have 90 days to finalize a contract with an external manager. If it doesn’t, or if the contract doesn’t meet the state’s guidelines, the state may pull the district’s accreditation, which would trigger dissolution of Adams 14.

State board chair Angelika Schroeder said no one wants to have to resort to that measure.

But districts should know, the state board does have “a few more tools in our toolbox,” she said.

In addition, if they get legal clearance, state board members would like to explicitly require the district:

  • To give up hiring and firing authority, at least for at-will employees who are administrators, but not teachers, to the external manager.
    When State Board member Steve Durham questioned the Adams 14 school board President Connie Quintana about this point on Wednesday, she made it clear she was not interested in giving up this authority.
  • To give up instructional, curricular, and teacher training decisions to the external manager.
  • To allow the new external manager to decide if there is value in continuing the existing work with nonprofit Beyond Textbooks.
    District officials have proposed they continue this work and are expanding Beyond Textbooks resources to more schools this year. The state review panel also suggested keeping the Beyond Textbooks partnership, mostly to give teachers continuity instead of switching strategies again.
  • To require Adams 14 to seek an outside manager that uses research-based strategies and has experience working in that role and with similar students.
  • To task the external manager with helping the district improve community engagement.
  • To be more open about their progress.
    The state board wants to be able to keep track of how things are going. State board member Rebecca McClellan said she would like the state board and the department’s progress monitor to be able to do unannounced site visits. Board member Jane Goff asked for brief weekly reports.
  • To allow the external manager to decide if the high school requires additional management or other support.
  • To allow state education officials, and/or the state board, to review the final contract between the district and its selected manager, to review for compliance with the final order.

Facing the potential for losing near total control over his district, Superintendent Javier Abrego Thursday afternoon thanked the state board for “honoring our request.”

The district had accepted the recommendation of external management and brought forward its own proposal — but with the district retaining more authority.

Asked about the ways in which the state board went above and beyond the district’s proposal, such as giving the outside manager the authority to hire and fire administrative staff, Abrego did not seem concerned.

“That has not been determined yet,” he said. “That will all be negotiated.”

The state board asked that the final order include clear instructions about next steps if the district failed to comply with the state’s order.

Indiana A-F grades

Why it’s hard to compare Indianapolis schools under the A-F grading system

PHOTO: Dylan Peers McCoy
Because Thomas Gregg Neighborhood School became an innovation school last year, the state uses a different scale to grade it.

A-F grades for schools across Indiana were released Wednesday, but in the state’s largest district, the grades aren’t necessarily an easy way to compare schools.

An increasing share of Indianapolis Public Schools campuses, last year about 20 percent, are being measured by a different yardstick than others, creating a system where schools with virtually identical results on state tests can receive vastly different letter grades.

The letter grades aim to show how well schools are serving students by measuring both how their students score on state tests and how much their scores improve. But as Chalkbeat reported last year, new schools and schools that join the IPS innovation network can opt to be graded for three years based only on the second measure, known as growth. Schools in the innovation network are part of the district, but they are run by outside charter or nonprofit operators.

Of the 11 out 70 Indianapolis Public Schools campuses that received A marks from the state, eight were graded based on growth alone. They included a school in its first year of operation and seven innovation schools.

At the same time, traditional neighborhood and magnet schools with growth scores as good as or better than the scores at A-rated innovation schools received Bs, Cs, and even Ds.

Of the 13 innovation schools that received grades for last school year, eight received As, two got Bs, two got Cs, and one got a D. Only Herron High School was graded on the same scale as other schools. (For high schools, grades incorporate other measures including graduation rates.)

The result is a system that most parents don’t understand, said Seretha Edwards, a parent of four children at School 43, a school that received a failing grade from the state but would have gotten a B if it were measured by growth alone.

“I just think it’s kind of deceiving,” she added. “I don’t think it paints a fair picture of the schools.”

Indianapolis Public Schools deputy superintendent for academics Aleesia Johnson said the growth scores show schools are on a good trajectory.

“If you see that kids are making progress in terms of growth, that’s a good sign that you’re on the right track,” she said.

Still, she acknowledged that “there’s still a lot of work to do” to get students to pass tests and show proficiency.

Johnson pointed out that often-changing standardized tests and different A-F grades can cause confusion for families, and those measures don’t provide a complete or timely picture for families who want to assess their schools or choose new ones. “I don’t think it gives a lot of valuable information,” she said.

Advocates have said the growth only model makes sense because schools shouldn’t be held accountable for the low passing rates of students that they just began educating. But in practice, the policy benefits charter and innovation schools, which enjoy strong support from Republican lawmakers.

“The concept behind the growth-only model was that we measured newer schools based off of what they are able to do for their students, rather than taking them where they received them,” said Maggie Paino, the director of accountability for the education department. “You’re taking strides to get toward proficiency.”

The situation is even more muddled than usual this year. Schools across the state received two letter grades. One was calculated under a state model that relies largely on test scores, and the other was determined under a plan the state uses to comply with federal standards.

In addition to helping parents choose schools, years of repeated low letter grades from the state can trigger intervention or takeover. But the state has deferred in decisions about intervening in low-rated schools to IPS in recent years.

Back in 2012, the state took over four chronically low-performing Indianapolis schools. Since Superintendent Lewis Ferebee took over, IPS has taken aggressive steps to overhaul struggling schools by “restarting” them as innovation schools with new managers. Other struggling schools have been closed.

School 63, which received its sixth consecutive F from the state, might have faced state intervention in the past. But the school is unlikely to face repercussions because IPS restarted the school by turning it over to an outside manager. The Haughville elementary school is now managed by Matchbook Learning.

Shaina Cavazos and Stephanie Wang contributed reporting.