study shows

The $100 million question: Did Newark’s school reforms work? New study finds big declines, then progress

PHOTO: TechCrunch/Creative Commons
Mark Zuckerberg

It was announced with much fanfare on Oprah in 2010: dramatic changes were coming to Newark’s schools, financed with $100 million from Facebook founder Mark Zuckerberg.

Those changes — including a new teachers’ contract and the expansion of charter schools — proved controversial and challenging to implement. But there hasn’t been a clear answer to the key question: Are students learning more now than they were then, thanks to the reform effort?

A new study, released Monday through the National Bureau of Economic Research, is among the first to try to answer.

It finds that by 2016, Newark students were making greater gains on English tests than they were in 2011. But the results are not uniformly positive. It finds no impact in math. And in both subjects, the reforms seem to have come with a cost: student achievement declined substantially in the first three years of the changes.

The study was funded by the Chan-Zuckerberg Initiative and conducted by a number of Harvard researchers, including Tom Kane, who said that the study’s results were independent of its funding source.

“This study confirms the progress that is being made in Newark schools,” Newark schools superintendent Chris Cerf said in a statement, “and shows that reforms undertaken — particularly in areas like citywide enrollment and expansion of high quality schools — are making a real difference for Newark students.”

Their findings have both educational and political import.

The 2010 announcement was heralded as a rare and powerful bipartisan alliance, with Democratic Mayor Cory Booker and Republican Governor Chris Christie united in their plan to make sweeping changes to Newark’s struggling schools and backed by Zuckerberg’s millions.

(The money — $100 million with another $100 million match — seemed eye-popping, though it only accounted for 4 percent of school spending in the city over the five years of the grant.)

The changes would include not only charter schools and performance-based pay for teachers, but also the closure of a number of district schools, and new enrollment system encompassing district and charter schools.

The initiatives soon came under scrutiny. Residents and students pushed back forcefully against some of the changes implemented by schools chief Cami Anderson. Ras Baraka, a school principal and sharp critic of Anderson, was elected mayor in 2014, beating Shavar Jeffries, who was more amenable to the reforms. Anderson was replaced by Cerf in June 2015.

Journalist Dale Russakoff wrote a largely critical account of changes that focused on how a large share of the Zuckerberg money went to high-paid consultants. Since, media reports have largely suggested that the approach failed and that the money was wasted.

Now, the central characters have largely moved on. But Zuckerberg and Booker, now a U.S. senator, are rumored to have national political ambitions, including potential runs for president in 2020.

Here are the key takeaways:

The overall effect of the reforms on student learning was mixed.

“By the fifth year of reform, Newark saw statistically significant gains in English and no significant change in math achievement growth,” the researchers conclude. “Perhaps due to the disruptive nature of the reforms, growth declined initially before rebounding in recent years.”

Source: “Assessing the Impact of the Newark Education Reforms”

The research, also released as a non-technical report, looks at two ways the reforms may have affected students: by making existing schools better and by moving students to more effective schools, including charters. Moving students to better schools did help, but existing schools didn’t consistently improve – and in the first three years got substantially worse in both subjects.

Something we still don’t know is what the cumulative impact of the reforms was on a student who attended Newark schools for five straight years. The study doesn’t answer that.

Another perhaps surprising finding, considering the common description of Newark schools as failing: The district had a growth rate before the changes that was about average for similar districts in New Jersey.

Students seemed to benefit from school closures.

The study finds students whose school was closed subsequently saw higher test score growth, particularly those moving into better schools. That’s consistent with other research. But Newark did not seem to close its worst schools, and even shut down a few schools that were average or above average. That might have limited the positive effects of closures.

Charter schools continued to outperform the district, but have grown less effective.

Part of the reform strategy was to expand Newark’s charter sector, since charter schools had been shown to substantially raise student test scores, relative to the district. The latest study found that charters continued to do better than the district, but the gap has essentially been cut in half.

That’s because charters’ effectiveness has decreased since 2011. It’s not clear why, but three times as many students attend charter schools in Newark now compared to 2010. That influx of new students and accompanying growing pains may be part of the explanation. 

The study also shows that charter schools serve different students than the district. Newark’s charter students are more likely to be African-American and female, and less likely to have a disability or limited proficiency in English.

The study comes with a few important caveats.

The spike in test-score growth toward the end of the five-year grant coincided with the introduction of a new test aligned with the Common Core, the PARCC. It also coincided with an increase in students opting out of state tests, both in Newark and statewide. The researchers try to account for this, but it’s not entirely clear if those changes skewed the findings.

Also, the researchers came to their conclusions by comparing test score growth of Newark’s students to students with similar backgrounds and in similar schools across New Jersey. That doesn’t guarantee that the study is able to isolate the effects of the reforms, but does allow for comparisons to places without the Zuckerberg money or attention.

The results don’t show whether the reforms “worked” — because that’s a complicated question.

The study is focused on standardized test scores, a significant limitation that means it doesn’t speak to other effects of the reforms on students. A separate analysis, funded by the Community Foundation of New Jersey and also released Monday, points out that high school graduation rates in Newark rose substantially in 2016 and 2017, after remaining flat between 2012 and 2015. Enrollment in the city schools has also trended upward in recent years. 

Source: “Moving Up: Progress in Newark’s Schools from 2010 to 2017”

The results also don’t account for political turmoil or the sense that the reforms were done to — rather than with — the community in Newark, whose schools had been under state control for a over two decades. An agreement was finalized in September to return them to community control.

“Ultimately we’re giving the parents the opportunity to have their democratic rights back,” Baraka told NPR, who argued in the same interview that the Zuckerberg dollars had not improved the school district. “There is no real kind of causal relationship between that money and the development of the traditional public schools in Newark.”

Gradebooks

Three Chicago principals and the war against Fs

If you’re a principal intent on disruption, here’s one place to start: Ban Fs.

“Fs and Ds are worthless,” Principal Juan Carlos Ocon told a group of rapt educators Thursday. The principal of Benito Juarez Community Academy in the predominantly Latino neighborhood of Pilsen spoke as part of a panel on improving student performance at a conference hosted by the University of Chicago Consortium on School Research.

The event took place during a daylong look at the consortium’s latest round of pivotal research, which draws a clear line from ninth grade performance to high school graduation.

Conferees discussed the latest data showing freshman GPAs in core classes — such as reading, math, and science — dropping a third of a point from their eighth-grade GPAs. One key finding: Failure in non-core classes, like PE, far exceeds similar eighth- grade numbers. But researchers didn’t uncover why as many Chicago freshmen fail PE as science. (Read more here.)

PHOTO: Cassie Walker Burke / Chalkbeat
Sarah Duncan, left, of the Network for College Success, moderates a panel on grades at a conference Oct. 11, on findings of the To & Through Project. Also appearing on the panel at the University of Chicago are Juan Carlos Ocon, Chad Adams, and Wayne Bevis.

Joined on the panel by fellow principals Chad Adams of Roger C. Sullivan High School in Rogers Park on the North Side and Wayne Bevis of Robert Lindblom Math and Science Academy, a test-in school in West Englewood, Ocon said he took a hard position to “ban Fs from kids’ lives.”

“It actually increases rigor,” he said, explaining how the mindset of his school has shifted from punitive deadlines to encouraging learning at a student’s pace. Any high schooler who isn’t proficient in a subject by June must keep going to class until the light bulb glows, Ocon said. “Our classes do not end in June when classes end in traditional high schools — our classes extend through second week of August.”

Panelists Adams and Bevis are also “blowing up” the idea of Fs. At Adams’ school, located in an immigrant-rich neighborhood and inside which 40 some languages are spoken, Fs aren’t quite verboten — but, every five weeks, teachers have to come clean with how many Fs they give.

“Teachers didn’t like it as first, but then they started to hold each other accountable,” Adams said. I have the same kids (as you do) in your class, but, look, I gave 4 Fs versus your 54. What are you doing?”

Bevis has done away with As through Fs entirely and moved to a numeric grading system that runs 1 to 4. He’s also implemented a buildingwide revision policy, which can be controversial at some schools. After receiving a grade, students have at least two weeks to resubmit revised work and show they have improved their skills. “Some teachers go longer than two weeks, up to a semester,” he said.

Though located in very different areas of the city, each school has seen significant gains in student performance, with consistent, year-over-year rises in graduation rates and “freshman on track” percentages — that is, the percentage of freshmen who are on track to graduate as measured at the end of ninth grade, a metric developed by the University of Chicago and a key measure of success in Chicago.

The principals used the panel session to share other practices they see improving performance in their schools.

At Lindblom, for example, a revolving weekly “colloquium class” offers students extra help in a particular subject. Students must submit requests by Monday night, and with input from teachers a computer spits out their assigned special class, which can change week-to-week. “There’s a consistent understanding among teachers and students that we need to target which skills they struggle with,” Bevis said.

At Juarez, teachers spent the past year studying and recommending a set of core developmental competencies, a list that includes perseverance and relationship skills. Daily lessons are built in during an advisory period, and the staff is on board since they helped create them, Ocon said.

Adams echoed the idea of building a high-performance culture starting with his teacher corps. He’s likewise building a set of core values to express what a Sullivan High School graduate represents. When it comes to creating a learning culture, staff buy-in is essential, he said. When it comes to change, “if the teachers aren’t ready, the kids won’t be ready.”

 

held back

Holding middle-schoolers back causes dropout rates to spike, new research finds

PHOTO: Seth McConnell/The Denver Post
A student opens his locker between classes at Overland Trail Middle School on August 17, 2017, in Brighton, Colorado. (Photo by Seth McConnell/The Denver Post)

To hold back or not to hold back? For many policymakers in the early 2000s, the answer was clear: it was time to stop allowing struggling students to keep moving through school.

“It’s absolutely insidious to suggest that a functionally illiterate kid going from third grade, it’s OK to go to fourth. Really?” explained Jeb Bush, the former governor of Florida, where he curtailed the practice known as social promotion.

Former New York City Mayor Michael Bloomberg felt the same way. He introduced a policy of holding back low-performing students and fired appointees to the city’s school board who pushed back in 2004.

The idea was that the stricter standards would help students and schools alike. More time in school would give students the chance to catch up, allowing them to avoid the academic failure that could result from being continually promoted with big gaps in their skills. Thousands of additional students in Florida, New York, and across the country were held back in line with that theory.

Now, enough time has passed to see what happened to some of those students years later — and two recent studies reach a decidedly dire conclusion.

Being held back a grade in middle school, researchers found, substantially increased the chance that students dropped out of high school. In Louisiana, being retained in either fourth or eighth grade increased dropout rates by nearly 5 points. In New York City, the spike was startling: dropout rates were 10 points higher than similar students who weren’t held back.

A policy meant to make sure students stay on track, then, appears to have caused more students to leave school altogether.

“The takeaway from this would be that, at a minimum, we should be retaining fewer middle school students,” said Paco Martorell, a professor at the University of California – Davis who studied the New York City policy.

“If we’re talking about a middle school policy, I would strongly suggest against that at this point,” said Marcus Winters, a professor at Boston University who studied the effects in Florida.

Whether retention ultimately helps or harms students remains a crucial question. Though some places have relaxed their policies, others are adopting stricter rules. Michigan’s new retention law, for one, threatens to ensnare the vast majority of Detroit’s third graders.

The research also offer some better news, including out of Florida. Holding back students when they are younger doesn’t have such clear negative effects. And summer school, which often goes along with retention, can help students, potentially outweighing the downsides of retention policies.

Here’s what else the new research tells us.

Retention seems to increase drop-out rates.

The latest studies focus on Louisiana, New York City, and Florida. Each compares similar students, some who just barely earned a passing score on a test and others who just missed the cut-off, allowing researchers to zero in on the effects of being held back.

In New York City, the grade retention policy initially seemed promising. A 2013 analysis showed that retained students scored higher on state tests when they eventually reached the next grade.

The latest study, released earlier this year by RAND, looks at the long-run effects for those students held back between 2004 and 2012 and paints a starkly different picture. Students who were held back in middle school were much more likely to drop out of high school than the students who also went to summer school but who moved to the next grade on schedule.

There were no clear effects for students held back in elementary school, according to that recent RAND study. (An older Chicago paper found something similar: retaining eighth-graders increased future dropout rates, but retaining sixth-graders had no clear effects.)

In Louisiana, the recent research found that retention increased high school dropout rates for fourth or eighth graders who were held back between 1999 and 2005.

The rules around retention vary widely. In most cases, students are held back after they fail to pass a test, sometimes after summer help. In Florida, policymakers focused their policy on third grade, but other places, like New York City, introduced strict holdover policies in a number of grades.

There’s also lots of variation in just how often students are held back. Nationally, about 2 percent of students are retained each year, a number that has held steady or modestly declined since the mid-1990s.

In New York City, only 1 percent of students were retained across a number of grades. But in Louisiana, about 7 percent of fourth-graders and 8 percent of eighth-graders were held back. When the policy was first introduced in Florida, around 13 percent of third-graders were kept back, a number that eventually fell to around 5 percent.

Helping students catch up over the summer is beneficial.

Another recent study offers better news: In Florida, retention of third-graders in the early 2000s had no effect on their high school graduation rates, and it actually improved students’ grades in high school. The study also found that retained students saw an immediate test-score bump, though that faded over time.

What explains the more positive results? It’s hard to know, because the Florida study looks at not just retention but a package of policies that went along with it, including summer school and assigning students in the repeated grade extra reading help.

The Louisiana paper may shed some light on this question. It was able to separate the consequences of being held back — which appear to be negative — from the consequences of going to summer school. Sending eighth graders to summer school decreased their chances of dropping out of school down the line and their likelihood of being convicted of a crime before their 18th birthday.

In other words, the different results suggest that being held back hurts students, but the summer support that goes along with it helps them.

Retention is costly, though perhaps less so than some think.

There’s another downside to holding students back: it’s expensive to pay to keep students in school for more time. It costs both the school system and the student, who potentially misses out on an extra year of earning as an adult.

“Being retained may not confer benefits that justify spending an additional year in the same grade,” the New York City researchers concluded. “This is especially true given our finding that retention entails significant financial costs.”

The New York City study finds that each retained student costs the system roughly an extra $2,600 — a large amount, though far less than annual per-student spending.

White students are more likely to avoid being held back.

The consequences of retention, good or bad, are disproportionately felt by some groups of kids.

For instance, in Louisiana 85 percent of retained students were black, even though black students represented less than half of students in the state’s public schools at the time. In New York City, black students were more than twice as likely to be retained as white students with similar test scores.

Nationally, black and Hispanic students are substantially more likely to be held back. Some of that can be tied to test scores, but other research shows that white, affluent families are particularly likely to circumvent policies around holding students back.

In Florida, children whose mothers did not hold a high school degree were 7 percentage points more likely to be retained compared to their peers with equal academic performance whose mothers were college educated, another study found. The students who moved ahead anyway often took advantage of exemptions, like portfolios created by teachers to demonstrate that students should move on to the next grade.

There’s still a lot we don’t know about the effects of retention.

Where does this new long-term research leave us?

Although retention itself may be harmful to students, the combination of retention and summer school in Florida and Louisiana was neutral or positive. One potential takeaway is that districts should maintain extra help for struggling students while scrapping retention.

But those policies are intimately connected in many places, so it’s not clear that you can pull out one part of the policy like a Jenga piece and have the rest of the apparatus remain intact. Indeed, new research by Winters, the Florida researcher, suggests that the threat of retention can cause students do better in school.

It might also spur changes across a school or community. That’s what is happening in Detroit, where the retention law has focused attention on young students’ reading. “We have to get involved now and do anything we can to get the proficiency level up for the second-graders,” as one Detroit principal told Chalkbeat in August.

Martorell, the Davis professor, says we still need more evidence to know whether there are hidden benefits to holding students back. But he warned that existing research indicates that some students are paying a price.

“Policymakers should think long and hard about whether these other effects that are not captured by these studies … are significant enough to incur monetary costs and potential negative effects on students,” he said.