Better curriculum was supposed to be one of the next big things in education.
In 2017, Bill Gates announced his influential foundation would invest fresh energy into helping create higher-quality classroom materials and get them into the hands of more teachers. A good curriculum, he said, “can improve student learning more than many costlier solutions.”
But new research, amounting to one of the largest-scale examinations of curriculum materials to date, suggests that the choice might not matter much — at least when it comes to elementary math test scores.
“We didn’t really find large differences in student achievement gains across different textbooks,” said the University of Maryland’s David Blazar, one of the study’s authors.
It’s a surprising conclusion for the six-state analysis, which was funded in part by the Bill and Melinda Gates Foundation and released Monday. And though it is just one study — which clashes with prior research — the results raise questions about the likelihood that the Gates Foundation’s curriculum-focused approach will boost student learning in the way its leaders hope.
“This study throws some cold water on the idea that just swapping out textbooks would make a big difference,” said Morgan Polikoff, a University of Southern California professor and another of the study’s researchers.
The Gates Foundation has said it wants to influence not just what materials schools choose, but how teachers are prepared to use them. In a statement, the foundation suggested that it is not rethinking its approach in response to the latest study.
“Other research suggests, and we believe, that when well used, high-quality curricula can enable more students to learn and succeed in their classes,” said Bob Hughes, the director of K-12 education at the Gates Foundation. “The challenge is to understand how the conditions, professional development, and types of materials combine to make this happen reliably.”
But the results were discouraging enough to dissuade the researchers from their plans to conduct annual follow-ups to help schools identify which curricula were coming out ahead.
“That is, of course, not where we ended up,” Blazar said. “We’re not going to be doing this again next year.”
The analysis focuses on fourth- and fifth-grade math materials used in thousands of schools across six states — California, Louisiana, Maryland, New Jersey, New Mexico, and Washington — in the Common Core era, school years 2014-15 through 2016-17.
(In addition to Gates, funders of the study included the Charles and Lynn Schusterman Foundation and the William and Flora Hewlett Foundation, which have also backed curriculum-improvement efforts. The Gates and Schusterman foundations are also funders of Chalkbeat.)
The researchers examined the 15 most common curricula in those states, including some that were not traditional textbooks, like the New York State–developed online curriculum EngageNY. They linked curriculum data with upper-elementary math test scores in each state to answer a key question: were certain curriculum choices consistently linked to students making bigger or smaller test score gains?
The answer was generally no. “No single text stands out as a consistent high- or low-performer in multiple states, nor in multiple years,” the researchers wrote.
Across the 15 options, only two (Houghton Mifflin Harcourt’s Math Expressions and McGraw-Hill’s Everyday Math) stood out in a positive way. Even then, those improvements disappeared when looking outside California.
“It was surprising,” said Blazar.
The results are markedly different from research on another aspect of a student’s classroom experience: teacher quality. Studies have found that teachers vary substantially in their effects on students’ test scores, attendance, and even income as adults.
One explanation for the results is that most curriculum options available to schools today are of roughly equal quality and touch on similar topics because many states have adopted shared academic standards — perhaps a victory of the Gates-pushed Common Core.
The study finds some evidence of that. Among schools that still used pre–Common Core curricula — about 16 percent of schools in 2016-17 — student achievement varied slightly more based on the textbook.
It’s difficult to know, though, exactly how much a chosen curriculum represents what students are actually exposed to in class. The vast majority of surveyed teachers reported using their textbook frequently, but said they regularly draw from outside sources, too.
“No teacher or school of teachers use a curriculum,” noted Darleen Opfer, who directs education research at the RAND Corporation. A precise answer to the question of whether curriculum affects student achievement would require knowing the full mix of materials teachers are using and how they are using them, she said.
“Without that level of detailed data on use and practice, we can’t understand whether curriculum matters for student achievement or not,” she said.
The study didn’t find any evidence that textbooks make more of a difference when they were consistently used by teachers, though. There was also no evidence that textbooks had more of an influence at schools where teachers got more training.
The study doesn’t look at the quality of that training, though. The Gates Foundation has also backed efforts to give teachers more and better preparation focused on how to best use their materials, and plans to announce up to $10 million in grants for such training this May.
The research also doesn’t directly account for other differences among curricula that matter for teachers and students, like whether the textbooks are engaging, whether they are culturally appropriate and represent students of different races, or whether they are easy for teachers to use.
So where does all this leave the push to improve America’s textbooks?
“I don’t think that the appropriate takeaway from this is necessarily that curriculum materials or textbooks don’t matter,” Polikoff said. “We have several studies that show that they do.”
What’s not obvious is why the new research looks so different. Some past studies used a higher-quality methodology, but were smaller; others focused on different grades or subjects. The recent study involves a much larger pool of schools and it is among the first to examine the issue after many districts adopted new textbooks because of the Common Core.
“Its conclusion is an important if disappointing one to ‘curriculum reform’ advocates like me,” said Michael Petrilli of the Fordham Institute, a conservative education think tank. But he argues the idea still holds promise, especially for helping newer and weaker teachers.
“A great curriculum — one that is aligned with standards, designed with teachers, and usable in real classrooms — can help teachers become more effective,” he said in an email. “At least if — and this is a big ‘if’ —those teachers receive lots of training, support, coaching, and feedback regarding how to implement the program with their own students.”
Blazar said it would be good news if that kind of support for teachers made a difference. “But in order to decide whether that is an efficient policy strategy,” he said, “we need to understand how effective those implementation supports are — and at what cost.”