Sign up for Chalkbeat’s free weekly newsletter to keep up with how education is changing across the U.S.
KIPP, the country’s largest charter school network, touts its commitment to getting students to and through college, so a recent study offered a compelling test of its model.
Researchers compared students who won a random lottery to attend a KIPP middle school versus others who lost the lottery. The result: Both groups earned college degrees at similar rates: 22%. “KIPP middle schools had little to no impact on four-year degree completion rates,” researchers with the company Mathematica concluded.
But the story did not end there. Researchers then examined a subset of those students who also attended a KIPP high school. Although this was not based on a random lottery, these results were encouraging. Attending a KIPP middle and high school dramatically boosted students’ chances of entering and completing college.
This result was emphasized by KIPP officials and in news coverage of the study when it was released last month.
Put together, though, the findings offer a complex verdict for KIPP charter schools. The results for middle and high school combined are promising, and KIPP already has a track record of boosting student learning. But the less-encouraging middle-school-only results apply to a broader group of students and are more methodologically robust because they are based on a lottery.
That has some experts saying that more research is needed before firm conclusions about KIPP are made.
“This is suggestive evidence,” said Jon Baron, president for the Coalition for Evidence Based Policy, referring to the middle and high school results. “This really needs to be tested in further study before being accepted.” (Baron was previously an official at Arnold Ventures, the philanthropy that funded this research.)
The study examined over 2,000 students who applied to attend one of 21 KIPP middle schools in 2008, 2009, or 2011. Researchers tracked and compared lottery winners versus losers for the next decade plus.
Prior research has found that KIPP improves test scores, but this new study showed limited longer-term benefits from attending a KIPP middle school. While there was some hint that students were more likely to enroll in college, they persisted and graduated college at similar rates as students who lost a KIPP lottery. This was the study’s “primary analysis.”
But then the Mathematica researchers undertook a further “exploratory analysis.” This is researchers’ way of saying that they are a bit less confident in these findings. Here, the study used an approach that is rigorous, but not based on a random lottery. (Nevertheless, a number of news outlets and commentators inaccurately described these exploratory findings as lottery-based.)
This secondary analysis focused on a smaller group of KIPP middle school students who also attended a KIPP high school. Those students appeared to benefit quite a lot from the longer KIPP experience: 39% of them earned a four-year college degree, compared to 20% of a comparison group.
These results, which were distributed to reporters last month ahead of the release of the study, were touted by KIPP leaders.
“The Mathematica study shows that a continuous KIPP education, spanning middle school and high school, is life-changing and would essentially close the educational-opportunity gaps facing Black and Latinx students,” said KIPP foundation CEO Shavar Jeffries in a statement.
Mathematica researchers say the gains may be due to the focus of KIPP high schools in getting its students into college. “It is possible that this large effect results from combining the well-established benefits of attending a KIPP middle school (a substantial boost to students’ academic achievement) with the strong emphasis on college-related supports found in KIPP high schools,” the study says.
The two findings create something of a puzzle, though: the group of KIPP middle and high school students who saw big gains in college completion were a subset of the larger group of middle school students who did not experience any improvements. And yet the overall effect on college completion for middle school was estimated to be close to zero. This is possible because the middle and high school group was a small subset of the whole sample, said Ira Nichols-Barrer, one of the Mathematica researchers.
He said that further research would help explain the disconnect between the study’s two big findings. “Our hope is that this is not the last phase of this study,” he said.
Matt Barnum is interim national editor, overseeing and contributing to Chalkbeat’s coverage of national education issues. Contact him at mbarnum@chalkbeat.org.
