Summit Learning declined to be studied, then cited collaboration with Harvard researchers anyway

Summit Learning, a fast-growing “personalized learning” system, touts a partnership with Harvard researchers even though Summit actually turned down their proposal to study the model.

The online platform is backed by Facebook founder Mark Zuckerberg’s philanthropy and is now being used in 380 schools across the U.S.

The program “is based on collaborations with nationally acclaimed learning scientists, researchers and academics from institutions including the Harvard Center for Education Policy Research,” Summit’s website says. “Summit’s research-backed approach leads to better student outcomes.” Schools have used that seeming endorsement to back up their decision to adopt the model.

In fact, though, there is no academic research on whether Summit’s specific model is effective. And while Summit helped fund a study proposal crafted by Harvard researchers, it ultimately turned them down.

“They didn’t tell us explicitly why,” said Tom Kane, a Harvard education professor and faculty director of the Center for Education Policy Research. “All I can say is that the work that we did for Summit involved planning an evaluation; we have not measured impacts on student outcomes.”

Summit’s founder Diane Tavenner said the organization had a number of reasons for not moving forward with the proposed study, including its potential to burden teachers and to limit the platform’s ability to change or grow. Their general approach is backed by other research, she said, and their track record as a charter network.

As to the mention of the Harvard center on Summit’s website, Tavenner said the organization had learned a lot from the process of developing a potential study. Tavenner said that, after Chalkbeat began reporting this story, she offered to change the website’s language, but said Kane had not asked her to do so.

More broadly, Tavenner says she is skeptical of the usefulness of large-scale research of the sort the Harvard team proposed, saying the conclusions might be of interest to journalists and philanthropists, not schools.

“I’m not willing to give up what’s best for kids for those two audiences,” Tavenner told Chalkbeat last month.

It’s a notable stance for Summit, given its ambitious claims and the platform’s wide reach.

As “personalized learning” becomes a more popular idea among those trying to improve America’s schools, Summit’s platform has been adopted for free by schools across the country. That’s thanks largely to the backing of the Chan Zuckerberg Initiative, the philanthropy poised to receive Zuckerberg’s billions. Summit’s model has drawn praise from parents and teachers in some schools, but proven controversial in others.

Regardless, CZI’s support means Summit could continue to grow rapidly — which has some observers wondering when its backers will show that what it’s offering is particularly effective.

“I do think that there is an obligation to provide credible evidence to schools when you’re trying to convince them to adopt things,” said John Pane, a researcher at the RAND Corporation who has extensively studied personalized learning initiatives.

Summit spreads, but research talks with Harvard team fizzle

Summit’s claims about a Harvard collaboration have their roots in conversations that began in  late 2016.

Zuckerberg’s wife, pediatrician Priscilla Chan, took a fateful tour of a school in the Summit Public Schools charter network two years earlier. The network soon began working with a Facebook engineering team to build out its technology.

Summit’s model has a number of components: a curriculum in core subjects for grades four through 12; weeks scheduled for students to deeply examine a topic of interest; long-term mentors for students; and a technology platform, which serves as the approach’s organizing structure. The goal is to better engage students and to give them more control over what and how they learn, Summit says.

By the 2016-17 school year, Summit had rolled out its program to more than 100 schools outside its own network. That’s also about when Summit started talks with Harvard professors Marty West and Kane.

An ideal study might have randomly assigned schools or students to use the learning platform, creating two groups that could be compared. That was a non-starter for Tavenner, as it would limit schools’ access to the platform. If 250 schools were assigned to use it, and another 250 expressed interest but were not, for example, that would be bad for students, she said last month while discussing the organization’s approach to research.

“Am I really going to say to 250 people, ‘You know what, we’re not going to actually help you, even though we actually could right now?’” she said.

Kane says they came up with a few alternatives: comparing students using Summit to others not using it in the same school or comparing schools that had adopted Summit to similar schools that hadn’t. They suggested tracking test scores as well as suspensions and attendance, measuring the effectiveness of the support offered to teachers, and using surveys to measure concepts important to Summit, like whether students felt in control of their schoolwork.

But Summit passed on an evaluation. “After many conversations with Harvard and the exploration of multiple options, we came to recognize that external research would need to meet certain baseline criteria in order for us to uphold in good faith our partnership with schools, students, and parents,” Tavenner said.

Metrics were a particular concern. “Standardized tests are not good measures of the cognitive skills,” a Summit spokesperson said, saying the organization had developed better alternatives. “Attendance and discipline are not measures of habits of success, full stop.” Tavenner said she feared that a study could stop Summit from being able to make changes to the program or that it might stop participating schools from adding new grades. (Kane and West say their plan wouldn’t have limited growth or changes.)

Tavenner told Chalkbeat that research of the kind the Harvard team was offering isn’t needed to validate their approach. Summit is based on decades of research on ideas like project-based learning, she said, citing the organization’s report titled “The Science of Summit.”

Dan Willingham, a University of Virginia educational psychologist, said that’s useful, but not the same as knowing whether a specific program helps students.

“You take a noticeable step down in confidence when something is not research-based but rather research-inspired,” he said, while noting that many education initiatives lack hard evidence of success. “There’s a hell of a lot going on in education that’s not being evaluated.”

What about Summit’s original charter network, now 11 schools? Summit cites internal data showing its graduates have success being accepted to college. But outside research is limited. A 2017 study by the Stanford-based group CREDO found that attending Summit led to modest declines in students’ reading scores and had no clear effect in math, though it looked at only a small portion of the network’s students.

The Summit charter schools are also part of an ongoing study of economically integrated charter schools, and a few were included in two widely cited RAND studies looking at personalized learning, though they didn’t report any Summit-specific information. California’s notoriously limited education data access has stymied more research, Tavenner said.

What does philanthropy owe the public?

Today, Summit’s learning platform has far outpaced its charter network. About 380 schools, with over 72,000 students, use the platform; the national charter network KIPP, by comparison, runs 224 schools serving around 100,000 students.

Summit now gets its engineering help from the Chan Zuckerberg Initiative, not Facebook. That philanthropic partnership has fueled its growth: While CZI has not disclosed how much it’s given to Summit, the Silicon Valley Community Foundation — through which CZI funnels much of its education giving — lists grants to Summit totalling over $70 million in 2016 and 2017.

Summit has also netted $2.3 million for the platform from the Bill and Melinda Gates Foundation in 2016, and another $10 million in 2017. (CZI, the Gates Foundation, and the Silicon Valley Community Foundation are all funders of Chalkbeat.)

Some major foundations regularly invest in research to better understand whether their gifts are doing good, noted Sarah Reckhow, a Michigan State professor who studies education philanthropy. In a number of instances, that research comes to unfavorable conclusions, like a Gates-funded study on its teacher evaluation initiative or a Walton Family Foundation-backed evaluation of charter schools’ propensity to screen out students with disabilities. (A Gates spokesperson said that part of its $10 million to Summit was set aside for “measurement and evaluation.”)

Reckhow said she hasn’t yet seen that same inclination from CZI. And she worries that school districts might be less likely to carefully examine programs that are offered free of charge, like Summit.

“If you reduce that barrier, you’re making it potentially more likely to adopt something without as much scrutiny as they otherwise might do,” she said. “That increases the obligation of Summit and CZI to evaluate the work.”

CZI spokesperson Dakarai Aarons said the organization is committed to research and to Summit, and pointed to a number of schools and districts that saw academic improvements after introducing Summit’s platform. “As the program grows, we look forward to expanded research to help measure its long-term impact,” he said.

Tavenner said Summit is exploring other options to prove its approach is working, including talking to researchers who study continuous improvement. “We can’t just keep saying no to [randomized studies],” she said. “We’ve got to have another way, but I don’t have another way yet.”

Researchers Kane and West, for their part, say Summit’s concerns about evaluating its evolving model should also raise questions about Summit’s swift spread.

“The evaluation we proposed would have assessed the impact of the model at that point in time, even if the model continued to evolve,” they wrote in an email. “When a model is still changing so radically that a point in time estimate is irrelevant, it is too early to be operating in hundreds of schools.”

“Unfortunately, Summit is closer to the rule than the exception,” they said.