where's the research

Summit Learning declined to be studied, then cited collaboration with Harvard researchers anyway

English teacher Adelaide Giornelli works with ninth grade students on computers at Shasta charter public high school, part of the Summit public school system. (Photo by Melanie Stetson Freeman/The Christian Science Monitor via Getty Images)

Summit Learning, a fast-growing “personalized learning” system, touts a partnership with Harvard researchers even though Summit actually turned down their proposal to study the model.

The online platform is backed by Facebook founder Mark Zuckerberg’s philanthropy and is now being used in 380 schools across the U.S.

The program “is based on collaborations with nationally acclaimed learning scientists, researchers and academics from institutions including the Harvard Center for Education Policy Research,” Summit’s website says. “Summit’s research-backed approach leads to better student outcomes.” Schools have used that seeming endorsement to back up their decision to adopt the model.

In fact, though, there is no academic research on whether Summit’s specific model is effective. And while Summit helped fund a study proposal crafted by Harvard researchers, it ultimately turned them down.

“They didn’t tell us explicitly why,” said Tom Kane, a Harvard education professor and faculty director of the Center for Education Policy Research. “All I can say is that the work that we did for Summit involved planning an evaluation; we have not measured impacts on student outcomes.”

Summit’s founder Diane Tavenner said the organization had a number of reasons for not moving forward with the proposed study, including its potential to burden teachers and to limit the platform’s ability to change or grow. Their general approach is backed by other research, she said, and their track record as a charter network.

As to the mention of the Harvard center on Summit’s website, Tavenner said the organization had learned a lot from the process of developing a potential study. Tavenner said that, after Chalkbeat began reporting this story, she offered to change the website’s language, but said Kane had not asked her to do so.

More broadly, Tavenner says she is skeptical of the usefulness of large-scale research of the sort the Harvard team proposed, saying the conclusions might be of interest to journalists and philanthropists, not schools.

“I’m not willing to give up what’s best for kids for those two audiences,” Tavenner told Chalkbeat last month.

It’s a notable stance for Summit, given its ambitious claims and the platform’s wide reach.

As “personalized learning” becomes a more popular idea among those trying to improve America’s schools, Summit’s platform has been adopted for free by schools across the country. That’s thanks largely to the backing of the Chan Zuckerberg Initiative, the philanthropy poised to receive Zuckerberg’s billions. Summit’s model has drawn praise from parents and teachers in some schools, but proven controversial in others.

Regardless, CZI’s support means Summit could continue to grow rapidly — which has some observers wondering when its backers will show that what it’s offering is particularly effective.

“I do think that there is an obligation to provide credible evidence to schools when you’re trying to convince them to adopt things,” said John Pane, a researcher at the RAND Corporation who has extensively studied personalized learning initiatives.

Summit spreads, but research talks with Harvard team fizzle

Summit’s claims about a Harvard collaboration have their roots in conversations that began in  late 2016.

Zuckerberg’s wife, pediatrician Priscilla Chan, took a fateful tour of a school in the Summit Public Schools charter network two years earlier. The network soon began working with a Facebook engineering team to build out its technology.

Summit’s model has a number of components: a curriculum in core subjects for grades four through 12; weeks scheduled for students to deeply examine a topic of interest; long-term mentors for students; and a technology platform, which serves as the approach’s organizing structure. The goal is to better engage students and to give them more control over what and how they learn, Summit says.

By the 2016-17 school year, Summit had rolled out its program to more than 100 schools outside its own network. That’s also about when Summit started talks with Harvard professors Marty West and Kane.

An ideal study might have randomly assigned schools or students to use the learning platform, creating two groups that could be compared. That was a non-starter for Tavenner, as it would limit schools’ access to the platform. If 250 schools were assigned to use it, and another 250 expressed interest but were not, for example, that would be bad for students, she said last month while discussing the organization’s approach to research.

“Am I really going to say to 250 people, ‘You know what, we’re not going to actually help you, even though we actually could right now?’” she said.

Kane says they came up with a few alternatives: comparing students using Summit to others not using it in the same school or comparing schools that had adopted Summit to similar schools that hadn’t. They suggested tracking test scores as well as suspensions and attendance, measuring the effectiveness of the support offered to teachers, and using surveys to measure concepts important to Summit, like whether students felt in control of their schoolwork.

But Summit passed on an evaluation. “After many conversations with Harvard and the exploration of multiple options, we came to recognize that external research would need to meet certain baseline criteria in order for us to uphold in good faith our partnership with schools, students, and parents,” Tavenner said.

Metrics were a particular concern. “Standardized tests are not good measures of the cognitive skills,” a Summit spokesperson said, saying the organization had developed better alternatives. “Attendance and discipline are not measures of habits of success, full stop.” Tavenner said she feared that a study could stop Summit from being able to make changes to the program or that it might stop participating schools from adding new grades. (Kane and West say their plan wouldn’t have limited growth or changes.)

Tavenner told Chalkbeat that research of the kind the Harvard team was offering isn’t needed to validate their approach. Summit is based on decades of research on ideas like project-based learning, she said, citing the organization’s report titled “The Science of Summit.”

Dan Willingham, a University of Virginia educational psychologist, said that’s useful, but not the same as knowing whether a specific program helps students.

“You take a noticeable step down in confidence when something is not research-based but rather research-inspired,” he said, while noting that many education initiatives lack hard evidence of success. “There’s a hell of a lot going on in education that’s not being evaluated.”

What about Summit’s original charter network, now 11 schools? Summit cites internal data showing its graduates have success being accepted to college. But outside research is limited. A 2017 study by the Stanford-based group CREDO found that attending Summit led to modest declines in students’ reading scores and had no clear effect in math, though it looked at only a small portion of the network’s students.

The Summit charter schools are also part of an ongoing study of economically integrated charter schools, and a few were included in two widely cited RAND studies looking at personalized learning, though they didn’t report any Summit-specific information. California’s notoriously limited education data access has stymied more research, Tavenner said.

What does philanthropy owe the public?

Today, Summit’s learning platform has far outpaced its charter network. About 380 schools, with over 72,000 students, use the platform; the national charter network KIPP, by comparison, runs 224 schools serving around 100,000 students.

Summit now gets its engineering help from the Chan Zuckerberg Initiative, not Facebook. That philanthropic partnership has fueled its growth: While CZI has not disclosed how much it’s given to Summit, the Silicon Valley Community Foundation — through which CZI funnels much of its education giving — lists grants to Summit totalling over $70 million in 2016 and 2017.

Summit has also netted $2.3 million for the platform from the Bill and Melinda Gates Foundation in 2016, and another $10 million in 2017. (CZI, the Gates Foundation, and the Silicon Valley Community Foundation are all funders of Chalkbeat.)

Some major foundations regularly invest in research to better understand whether their gifts are doing good, noted Sarah Reckhow, a Michigan State professor who studies education philanthropy. In a number of instances, that research comes to unfavorable conclusions, like a Gates-funded study on its teacher evaluation initiative or a Walton Family Foundation-backed evaluation of charter schools’ propensity to screen out students with disabilities. (A Gates spokesperson said that part of its $10 million to Summit was set aside for “measurement and evaluation.”)

Reckhow said she hasn’t yet seen that same inclination from CZI. And she worries that school districts might be less likely to carefully examine programs that are offered free of charge, like Summit.

“If you reduce that barrier, you’re making it potentially more likely to adopt something without as much scrutiny as they otherwise might do,” she said. “That increases the obligation of Summit and CZI to evaluate the work.”

CZI spokesperson Dakarai Aarons said the organization is committed to research and to Summit, and pointed to a number of schools and districts that saw academic improvements after introducing Summit’s platform. “As the program grows, we look forward to expanded research to help measure its long-term impact,” he said.

Tavenner said Summit is exploring other options to prove its approach is working, including talking to researchers who study continuous improvement. “We can’t just keep saying no to [randomized studies],” she said. “We’ve got to have another way, but I don’t have another way yet.”

Researchers Kane and West, for their part, say Summit’s concerns about evaluating its evolving model should also raise questions about Summit’s swift spread.

“The evaluation we proposed would have assessed the impact of the model at that point in time, even if the model continued to evolve,” they wrote in an email. “When a model is still changing so radically that a point in time estimate is irrelevant, it is too early to be operating in hundreds of schools.”

“Unfortunately, Summit is closer to the rule than the exception,” they said.

upheaval

Frustrations over principal turnover flare up at IPS School 43

PHOTO: Dylan Peers McCoy
School 43

It began with a tame slideshow presentation about hiring a new principal at School 43. But the Wednesday night meeting soon spiraled into a venting session — as parents and teachers pleaded with Indianapolis Public Schools to send in more staff.

Bakari Posey, the principal of School 43, departed for another job last week in the latest upheaval at the school, which is also known as James Whitcomb Riley. The assistant principal, Endia Ellison, has taken over in an interim capacity, as the district searches for a new leader for the school, which has faced significant turnover in recent years.

“This school needs help,” said Natasha Milam, who has three children at School 43, which serves about 450 students in prekindergarten to eighth-grade. “We need you all to listen. And we need you all to hear us.”

Milam, who volunteers at the school, said that because the building does not have enough staff to handle behavior problems, students are suspended far too often — meaning students are at home doing chores or getting into trouble, instead of in class learning.

Many in the neighborhood had hoped Posey, who is from the community, would be able to turn the school around after the previous two school leaders left their posts just months into the job. But under Posey’s leadership, the school continued to struggle on state tests, with just 7 percent of students passing both the math and English exams last year.

And after two-and-a-half years on the job, Posey left and began working this week as assistant principal at Fall Creek Valley Middle School in Lawrence Township. In an email Thursday, Posey said that he left because he thought the position in Lawrence would help him grow professionally and it was closer to his home.

Posey also disputed the picture of School 43 as a campus in crisis. He said this school year, there hasn’t been “turmoil in the school in regards to student behavior,” suspensions were down, and the campus has been “very calm.” (Suspension numbers could not immediately be verified.) He also said that Indianapolis Public Schools provided “great support” to school staff.

Nonetheless, parents and teachers’ at the meeting Wednesday said the school has serious problems.

Ryesha Jackson, a 4th-grade teacher who has been at the school a little over a year, said there are not enough staff to help with student discipline problems. That makes it hard for educators to teach, she said.

“We have fights almost every day,” Jackson said. “I guess my question is, ‘What are we doing right now to support teachers?’”

School 43 is a neighborhood school, on the north side of the district. More than 75 percent of students there are black, and almost 70 percent are poor enough to qualify for free or reduced-price meals — about the district average.

Indianapolis Public Schools interim Superintendent Aleesia Johnson said district and school leaders would work together to develop a plan to address the urgent problems at School 43.

“But what I can’t give you right now is the plan for that help,” she said. “That takes time and coordination with the school staff.”

The district is gathering input about what school community members are looking for in a principal before posting a listing, officials said. Finalists will be interviewed by committees of parents, community members, and school and district staff. The goal is to name a new principal by April.

Also at Wednesday’s meeting was a small contingent from the IPS Community Coalition, a group that is often critical of the Indianapolis Public Schools administration, particularly the district’s partnerships with charter schools.

Michele Lorbieski, a resident from the north side who ran unsuccessfully for the Indianapolis Public Board with the support of the coalition last year, said the district cannot just rely on the next principal to fix the school.

“What I’d hoped to hear tonight was what the school district was doing to put things in place to stop this revolving door of principals,” she said.

District officials did not directly address why turnover has been so high among principals at School 43. But Brynn Kardash, a district official who recently began working with the school, said that the central office is doing more to support it this year.

School 43 was added this year to the transformation zone — an effort to help troubled schools that includes dedicated support and regular visits from a team at the central office, said Kardash, the district’s executive director of schools for the zone. Educators in the zone get additional training, extra planning time, and help analyzing student data, she said.

“The goal is to really support Ms. Ellison in work that she’s doing,” Kardash said, “which then leads to, hopefully, teachers feeling that support in the classroom.”

technical difficulties

This personalized learning program was supposed to boost math scores. It didn’t, new study finds

PHOTO: Patrick Wall
A student at I.S. 228 in Brooklyn does online work through Teach to One, a program that grew out of the iZone.

A program that Bill Gates once called “the future of math” didn’t improve state test scores at schools that adopted it, according to a new study.

The research examines Teach to One, a “personalized learning” program used in schools across 11 states and which has drawn support from a number of major funders, including the Gates Foundation, Chan Zuckerberg Initiative, and Netflix co-founder Reed Hastings. (Gates and CZI are also funders of Chalkbeat.)

At five schools in Elizabeth, New Jersey, students who used Teach to One didn’t improve any faster than similar students who didn’t use the program, even after three years. The results underscore the limited evidence for claims that such technology programs can dramatically improve student learning, even as they have become magnets for philanthropic dollars.

“The original aspirations, that Teach to One programs were going to have huge positive effects on math scores — we can rule that out with these studies,” said Jonah Rockoff, a Columbia professor who studied an earlier iteration of the program.

Teach to One says its approach is designed to help students steadily learn math skills, regardless of how unprepared or advanced they are. Students spend time on a computer as well as with a teacher and working in small groups. Students receive individualized schedules each day based on their progress, and a computer program adapts the curriculum to students’ strengths and weaknesses in the form of a “playlist.”

New Classrooms, the organization behind Teach To One, suggests that the Elizabeth results aren’t the full story.

It points to a separate analysis released this week that looks at a broader group of schools — 14, from several districts — that used the program. That study shows Teach to One students making above-average gains on a test known as the MAP, which is taken on a computer with questions changing as students answer correctly or incorrectly.

New Classrooms co-founder Joel Rose suggested in a statement that those computer-adaptive tests capture something that state tests can miss: students’ progress.

“What seems to be emerging is a real tension in math between approaches focused on long-term academic growth and state accountability systems,” he said.

Rockoff said there might be something to New Classroom’s argument that the study using adaptive test is better able to showcase students’ gains. “If [students] are at a grade four level but they’re in grade six, teaching them grade four material is going to hurt them on the state test,” he said.

But the author of the second study, Jesse Margolis, and a number of other researchers who spoke to Chalkbeat note that it cannot show whether Teach to One caused any of the students’ gains, though — a major limitation.

“While this study cannot establish causality, it is encouraging,” Margolis wrote. (The New Jersey study is better able to establish cause and effect, but it also has limitations and does not rely on random assignment.)

The New Jersey study isn’t the first to show that Teach to One didn’t improve test scores: so did Rockoff’s 2015 report on three New York City middle schools that looked at both state and MAP tests.

One possible explanation is that Teach to One is helpful to students in some places but not others. Margolis said his study examined the same five Elizabeth schools as the Columbia study and also found minimal gains there, but that schools elsewhere seemed to see larger improvements.

Researcher John Pane of RAND, a leader in studying personalized learning, says the results are important to understanding a field with limited research to date.

“Because we have so little evidence on personalized learning,” he said, “every data point can be helpful for us to start triangulating and piecing together what works and what doesn’t work.”