Mayor Bill de Blasio’s big bet on pumping millions of dollars into low-performing schools instead of closing them down is creating “meaningful” academic benefits.
That’s according to a forthcoming report from an unlikely source: the conservative-leaning Manhattan Institute, a frequent critic of the mayor’s education policies. The report is the first independent analysis to show that the Renewal program is producing academic benefits, as measured by math and reading scores on state tests.
School Renewal — a program that infused 94 of the city’s lowest-performing schools with additional social services, nonprofit partnerships and academic support — tended to produce larger academic gains than the rest of the city’s schools, including those that had similarly low test scores before the program began.
In all, the report estimates, Renewal boosted student achievement by a “meaningful magnitude” — the equivalent of about 49 days of extra instruction in reading and 33 days in math, as measured by gains in state test scores. (The report focuses only on elementary and middle schools. It does not look at academic progress in Renewal high schools, which comprise a little more than a third of the program.)
“The evidence I find is that that the schools are better than they would have been without the label,” said Marcus Winters, the study’s author.
That finding is good news for an administration that has struggled to point to rigorous statistical evidence that its plan for low-performing schools is creating clear progress. It also comes at a key moment, as the program is nearing the end of the third year of what de Blasio initially described as a three-year program (though it is slated to continue next year).
Still, the Manhattan Institute’s relatively positive findings are complicated by a recent analysis conducted by Teachers College professor Aaron Pallas. Using a different statistical model and an additional year of data, Pallas found that Renewal schools generally did not make bigger gains in reading or math when paired with other struggling schools that didn’t receive extra support.
“There’s some hint of progress in raising test scores that wasn’t present in my analysis,” Pallas said, noting the Manhattan Institute analysis focused on one year of data. “It’s not, I think, a strong case.”
In its response to the report, the city seemed reluctant to seize on its specific findings as evidence of the program’s success, pointing instead to general increases in achievement at Renewal schools. “Renewal Schools are making real progress,” education department spokesman Michael Aciman said in a statement. “Across the program, graduation rates are up, chronic absenteeism is down, state test scores are improving, and teachers are developing stronger instructional practices.”
To assess the program, Winters used a model that measured yearly changes in test scores in 2015 and 2016 at Renewal schools compared with those outside the program. The Renewal program had the strongest effect in 2016, the year after it was fully implemented, boosting math and reading scores by a statistically significant margin overall (though the positive benefits were not consistent across all grade levels).
The positive effect becomes less pronounced when Renewal schools are compared to others that also had low scores, or when the timeline is extended to include 2015 data from when the program was still rolling out.
Jonah Rockoff, an education researcher at Columbia University who reviewed the Manhattan Institute report and Pallas’s data, cautioned against claiming the new report offers definitive evidence of the Renewal program’s effect on test scores.
“The [Manhattan Institute] report is more upbeat on Renewal schools raising test scores,” he said. “My sense is that any positive effects are quite small.”
Winters also claims that the Renewal program — which has cost roughly $386 million so far — is both more costly and less effective than former Mayor Michael Bloomberg’s approach of giving schools letter grades and closing ineffective ones.
Under Bloomberg’s letter grade system, for instance, schools that received an ‘F’ improved the following year by the equivalent of 36 days of learning in reading and 72 days in math, the report says, more than schools improved under Renewal. But the study does not include a rigorous cost-benefit analysis of Bloomberg’s versus de Blasio’s efforts to improve struggling schools — and city officials said the report’s comparisons are unfair.
“The A-F grading system does not take into account the systemic challenges that many Renewal schools faced when they entered the program,” Aciman said. “The study overestimates the impact that a single letter grade can have on improving a school, and underestimates the impact of the research-based supports that are being implemented at Renewal schools to improve classroom instruction, school climate and student performance.”
Correction: Due to errors in the original Manhattan Institute report, this story has been corrected to reflect the accurate number of extra days of instruction created by the Renewal program and Bloomberg’s accountability system.