Earlier this year, Brooklyn Assemblyman James Brennan issued a report arguing that the educational reforms formulated by Mayor Michael Bloomberg and Chancellor Joel Klein were not as influential as the policies and practices that preceded mayoral control. Although test scores have risen significantly over the past decade, the report claimed, most of the improvements predated mayoral control. Moreover, in some cases, the progress during the Bloomberg-Klein era has been slower than what was observed before. The report’s conclusions call into question claims about the impact of mayoral control, a hot topic for the next few months and perhaps beyond.
Andy Jacob, overworked and underpaid Department of Education spokesperson–I’m actually not kidding about this–countered with a memo seeking to refute Brennan’s report. The key claim: The test score gap between New York City and the rest of the state has closed more under mayoral control than it did before mayoral control, by a substantial margin. The source of the evidence? State tests in reading and math in both the fourth and eighth grades.
New York is a big and diverse state, and it’s not immediately obvious why comparing New York City to the state is a good idea. One rationale is that this comparison controls for year-to-year variation in the content or difficulty of the state exams. Thus, if the average test score gap between city and state were to shrink over time, we could discount the explanation that it’s just a function of the properties of the test.
As usual, skoolboy prefers to look at scale scores rather than the percentage of students who are proficient; the scale scores carry more information about student performance, and are less vulnerable to the distortion that high-stakes accountability can elicit. I looked at scale scores in fourth grade reading and math from 1999 to 2008, comparing the rate of improvement in New York City to growth across New York state overall. In particular, I examined whether the extent to which the gap between city and state closed over this period differed for 1999 to 2003, prior to mayoral control, and 2003 to 2008, during the Bloomberg/Klein era.
The picture is not as clean as Andy Jacob suggests. The first figure below shows trends in performance in 4th grade and 8th grade reading. The vertical line separates 1999-2003 from 2003-2008. In 4th-grade reading, the gap between New York City and New York state closed by 25% during 1999 to 2003, but only 10% between 2003 and 2008–a slower rate of closing the gap during the Bloomberg/Klein era. In 8th-grade reading, the story is worse: the gap between city and state increased by 2% between 1999 and 2003, but increased by an additional 16% between 2003 and 2008. Nothing for the DOE to crow about here!
The second figure shows trends in 4th grade and 8th grade math performance. Here, the DOE claims hold up. In 4th-grade math comparisons between city and state, the gap shrank by 43% between 1999 and 2003, and by 57% between 2003 and 2008-a higher rate of closure during the mayoral control era than before. Similarly, the city/state achievement gap in 8th-grade mathematics closed at a faster rate between 2003 and 2008 than it did between 1999 and 2003.
Unfortunately for Andy, he didn’t look at the National Assessment of Educational Progress data. New York City has been participating in the Trial Urban District Assessment (TUDA) since 2002, which generates NAEP scale scores for New York City and a number of other large urban districts that can be compared to the New York state NAEP scores.
In 2003–relatively early in the Bloomberg/Klein run–fourth-graders in New York City scored 12.3 points lower, on average, on the NAEP reading assessment, and 9.6 points lower, on average, on the NAEP math assessment, than fourth-graders in New York State overall. Although these gaps shrank by two or three points between 2003 and 2007, the shrinkage is not statistically significant. In other words, there is no persuasive evidence that the reading and math performance gap between fourth-graders in New York City and fourth-graders in New York state declined between 2003 and 2007, prime-time years for the Bloomberg/Klein reforms.
The picture is much the same at the eighth-grade level. New York City eighth-graders started out in 2003 13.5 points behind their state peers in reading, and 14 points behind in math. In 2007, the reading gap had actually increased by a point, and the math gap had shrunk by three points. Once again, though, these changes between 2003 and 2007 were not statistically significant. We cannot conclude that the reading and math performance gaps between New York City eighth-graders and eighth-graders across the state declined between 2003 and 2007.
Bottom line: On both the New York state reading test and the NAEP reading assessment, there is no evidence that the achievement gap between New York City and New York state overall closed faster during the Bloomberg/Klein era of mayoral control than the period preceding mayoral control. In mathematics, the city closed the gap faster under mayoral control than in the preceding period on the New York state math assessment, but the performance gap on the NAEP assessment has not shrunk appreciably during the Bloomberg/Klein administration.
About our First Person series:
First Person is where Chalkbeat features personal essays by educators, students, parents, and others trying to improve public education. Read our submission guidelines here.