A program that Bill Gates once called “the future of math” didn’t improve state test scores at schools that adopted it, according to a new study.
The research examines Teach to One, a “personalized learning” program used in schools across 11 states and which has drawn support from a number of major funders, including the Gates Foundation, Chan Zuckerberg Initiative, and Netflix co-founder Reed Hastings. (Gates and CZI are also funders of Chalkbeat.)
At five schools in Elizabeth, New Jersey, students who used Teach to One didn’t improve any faster than similar students who didn’t use the program, even after three years. The results underscore the limited evidence for claims that such technology programs can dramatically improve student learning, even as they have become magnets for philanthropic dollars.
“The original aspirations, that Teach to One programs were going to have huge positive effects on math scores — we can rule that out with these studies,” said Jonah Rockoff, a Columbia professor who studied an earlier iteration of the program.
Teach to One says its approach is designed to help students steadily learn math skills, regardless of how unprepared or advanced they are. Students spend time on a computer as well as with a teacher and working in small groups. Students receive individualized schedules each day based on their progress, and a computer program adapts the curriculum to students’ strengths and weaknesses in the form of a “playlist.”
New Classrooms, the organization behind Teach To One, suggests that the Elizabeth results aren’t the full story.
It points to a separate analysis released this week that looks at a broader group of schools — 14, from several districts — that used the program. That study shows Teach to One students making above-average gains on a test known as the MAP, which is taken on a computer with questions changing as students answer correctly or incorrectly.
New Classrooms co-founder Joel Rose suggested in a statement that those computer-adaptive tests capture something that state tests can miss: students’ progress.
“What seems to be emerging is a real tension in math between approaches focused on long-term academic growth and state accountability systems,” he said.
Rockoff said there might be something to New Classroom’s argument that the study using adaptive test is better able to showcase students’ gains. “If [students] are at a grade four level but they’re in grade six, teaching them grade four material is going to hurt them on the state test,” he said.
But the author of the second study, Jesse Margolis, and a number of other researchers who spoke to Chalkbeat note that it cannot show whether Teach to One caused any of the students’ gains, though — a major limitation.
“While this study cannot establish causality, it is encouraging,” Margolis wrote. (The New Jersey study is better able to establish cause and effect, but it also has limitations and does not rely on random assignment.)
The New Jersey study isn’t the first to show that Teach to One didn’t improve test scores: so did Rockoff’s 2015 report on three New York City middle schools that looked at both state and MAP tests.
One possible explanation is that Teach to One is helpful to students in some places but not others. Margolis said his study examined the same five Elizabeth schools as the Columbia study and also found minimal gains there, but that schools elsewhere seemed to see larger improvements.
Researcher John Pane of RAND, a leader in studying personalized learning, says the results are important to understanding a field with limited research to date.
“Because we have so little evidence on personalized learning,” he said, “every data point can be helpful for us to start triangulating and piecing together what works and what doesn’t work.”