When Mayor Bill de Blasio launched an aggressive and expensive campaign to turn around the city’s lowest performing schools, he made a big promise: Schools would see “fast and intense” improvements within three years.
Almost exactly three years later, and after flooding 78 schools with more than $386 million in new social services and academic support, there are signs that the Renewal program has generated gains in student learning. The evidence is based on two newly updated analyses of test score data — one from Marcus Winters, a fellow at the conservative-learning Manhattan Institute, and the other from Aaron Pallas, a professor at Teachers College.
But the researchers caution that those improvements are modest — when they exist at all — and don’t yet match the mayor’s lofty promises.
The results may have implications far beyond New York City, as a national and political test case of whether injecting struggling schools with resources is more effective than closing them.
The two researchers previously reviewed the first two years of test score data in elementary and middle schools in the Renewal program: Winters found a positive effect on test scores, while Pallas generally found little to no effect.
Now, as the program reaches its third birthday, the pair of researchers have updated their findings with new test score data from last school year, and largely reaffirmed their earlier conclusions.
“We’re not seeing large increases” in student achievement, Pallas said. “And the reality is it’s hard to get large increases in struggling schools.”
Some advocates have argued that it is too early to expect big shifts in test scores, and that infusing schools with extra social services like mental health counseling and vision screenings are valuable in themselves. But de Blasio’s promise of quick academic turnaround has invited questions about Renewal’s effectiveness and whether resources can be more effective in improving low-performing schools than shuttering them.
To assess the program’s academic effect, Pallas compared changes in Renewal school test scores to other schools that had similar test results and student demographics when the program started, but did not receive extra support.
The biggest gains Pallas found were concentrated at the elementary level.
Over the past three school years, 20 elementary schools in the Renewal program have made larger gains on average in math and reading than 23 similar schools that didn’t get extra resources. The proportion of elementary school students considered proficient in reading at Renewal schools increased from 7 percent in 2014 to 18 percent last year — an 11-point jump. Meanwhile, the comparison schools also saw gains, but only by seven percentage points, giving Renewal schools a four percentage point advantage.
At the middle school level, the results are less encouraging. The 45 Renewal middle schools did not collectively outperform a group of 50 similar schools outside the program in reading or math.
In math, for instance, Renewal school students improved from 5 percent proficient to 7 percent. However, the comparison schools outside the program improved by roughly the same margin — increasing proficiency from 6 to 9 percent (and still far below city average). In reading, Renewal middle schools showed slightly less growth than the comparison group.
City officials have argued that Pallas’ findings are misleading partly because Renewal schools and the comparison schools are not actually comparable. Renewal schools, they say, were designated based on a range of factors like school climate or teacher effectiveness, not just student demographics and test scores.
“The schools included in the study are neither similar nor comparable in quality and a comparison of the two dissimilar groups is unreliable at best,” Michael Aciman, an education department spokesman, said in a statement. Aciman added that Renewal schools have made larger gains in reading and math than similar schools across the state, and have made progress in reducing chronic absenteeism and improving instruction.
Pallas notes that there are some limitations to his approach, and acknowledges that he could not account for some differences between the two groups, such as the quality of a school’s principal. He also does not use student-level data, for instance, which would allow a more fine-grained analysis of whether the Renewal program is boosting student achievement. But Pallas, and other researchers who have previously reviewed his data, have said his model is rigorous.
The Manhattan Institute’s Winters found more positive trends than Pallas, consistent with his earlier findings. Using an approach that evaluates whether Renewal schools are outperforming historical trends compared with schools outside the program, Winters found that the Renewal program appeared to have a statistically significant effect on both reading and math scores — roughly equivalent to the difference in student achievement between charter schools and traditional district schools in New York City.
Asked about how to interpret the fact that his results tended to be more positive, Winters said either interpretation is plausible.
“It’s hard to tell which of these is exactly right,” he said. But “neither of us are finding results that are consistent with what we would expect if the program is having a large positive effect.”