Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

School choice

Denver judge blocks school transportation provision added to Colorado law

PHOTO: Denver Post file
Sam Boswell, 7, all bundled up in his winter clothes, splashes his way to the school bus on May 12, 2010.

A Denver judge struck down a provision of a bill related to the education of youth in foster care that would have removed barriers to transportation for all students.

The transportation provision was an amendment added by Republican lawmakers late in the 2018 session. Soon after the bill was signed by Gov. John Hickenlooper, several Colorado school districts and the associations that represent them filed a lawsuit to block it.

In a ruling issued Friday, Denver District Court Judge David Goldberg found that the amendment violated rules in the Colorado constitution that require every bill to have a clear title that explains what the bill is about and to deal only with one subject.

The bill’s title was “Improving Educational Stability for Foster Youth,” and it seeks to improve graduation rates for foster youth by requiring child welfare officials and school districts to work out transportation to the student’s home district when that’s in the child’s best interest. It also creates flexibility around graduation requirements when students do change schools. Foster youth have the lowest four-year graduation rates in the state, much lower even than homeless youth and students whose parents are migrant workers.

The tacked-on language was added in the Republican-controlled State Affairs committee five days before the end of the session. It said that a school board “may furnish transportation” to students who are enrolled in the district but who live in another district. The provision applies to all students, not just those who are in the foster system. It also struck language from an existing law that requires the consent of the school district from which students are being bused.

The amendment language came straight from a separate bill about expanding school choice that had been killed by Democrats in the House the day before.

Many school districts opposed the transportation provision because they feared it would open the door for better-off districts to poach students and undermine the meaning of school district boundaries. Advocates for school choice argued the provision was good policy that would allow more students, especially those from low-income families, take advantage of opportunities. They also argued, apparently unconvincingly, that it was required for implementation of the foster youth portions of the bill.

The Donnell-Kay Foundation intervened in the case in defense of the law. (The Donnell-Kay Foundation is a funder of Chalkbeat. You can read our ethics policy here.)

In his ruling, Goldberg said this specific issue has never been litigated in Colorado before, and he relied in part on rulings from other states with similar requirements. Bills with broad titles, he wrote, can be construed broadly and encompass a range of issues as long as they have some connection to the title. But bills with narrow titles must be construed narrowly — and this amendment didn’t make the cut.

“The subject of House Bill 18-1306 is out-of-home placed students and efforts to ensure educational stability,” Goldberg wrote, while the amendment’s subject “is all students, with no qualifiers, conditions, restrictions, or reference to out-of-home placed students. … House Bill 18-1306 seriously modifies transportation for all students and is hidden under a title relating exclusively to out-of-home placed students.”

Goldberg ruled that the amendment is “disconnected” from the rest of the bill, and neither lawmakers nor the public had enough notice about its inclusion before passage.

That leaves the rest of the foster youth bill intact and advocates for expanded school choice facing an uphill battle in a legislature in which Democrats, who are more likely to give priority to school district concerns, now control both chambers.

This isn’t an abstract issue. In 2015, more than 150 students who lived in the Pueblo 60 district but attended school in higher-performing Pueblo 70 lost access to transportation when the city-based district ordered its neighbor to stop running bus routes through its territory.

Online Shopping

Jeffco launches universal enrollment site to make school choice easy

PHOTO: Yesenia Robles, Chalkbeat
Students in a social studies class at Bear Creek High School in Jeffco Public Schools read about Genghis Khan.

Starting Monday, parents in Colorado’s second-largest district will be able to shop online for schools and, once enrollment opens in January, apply to as many as they like.

The launch of Enroll Jeffco, following the path paved by Denver Public Schools, means some 86,000 students and their parents won’t have to go to individual schools during the work day and fill out paper forms if they want to apply somewhere other than their neighborhood school.

The online system cost about $600,000 to develop and operate for this school year. The district expects it to cost about half of that annually going forward.

Universal enrollment systems allow parents to compare and apply to traditional district-run schools, district schools with specialized programming or models, known in Jeffco as options schools, and charter schools with a single application on the same website. Universal enrollment systems are a key component of what some call the “portfolio model,” in which districts oversee a range of school types and parents vote with their feet. They’ve been controversial in places, especially when coupled with aggressive school accountability policies that lead to school closures.

In Jeffco Public Schools, which is more affluent than many Denver metro area districts, officials see the move to a single, online enrollment system as a valuable service for parents.

“Regardless of how people feel about it, we operate in a competitive school choice environment, both inside the district and outside the district,” Superintendent Jason Glass said. “That compels us to make thinking about that transaction, making people aware of the options and enrolling in our schools, as frictionless and easy as possible.”

Colorado law requires schools in any district to admit any student for whom they have room and for whom the district can provide adequate services, after giving priority to students who live in the district. But many districts still require paper applications at individual schools, and schools in the same district might not have the same deadlines. A recent report by the conservative education advocacy group Ready Colorado found that parents who use school choice are more likely to be white, middle- or upper-class, and English-speaking than the state’s student population. The authors argue that districts should streamline the enrollment process and consider providing transportation to make choice more accessible.

Jeffco isn’t rolling out new transportation options yet, but it might use data from the enrollment process, including a parent survey that is built into the website, to see if that’s desired or feasible. And officials believe strongly that the new online enrollment system will open up more opportunities for low-income parents and those who don’t speak English.

The website will provide information in the district’s six most commonly spoken languages and should be optimized for use on mobile phones. All parents will be required to use the system to express their preferences, including the majority of parents who want to stay in their neighborhood school, and the district is planning significant outreach and in-person technical assistance.

We believe that if all parents are participating, it improves equity,” Glass said. “One of the things we struggle with is that upwardly mobile and affluent parents tend to be the ones who take advantage of school choice. We want all of our schools to be available to all of our families. We think being able to search through and make the enrollment process as easy as possible is an equity issue.”

But critics of universal enrollment systems worry that the ease of application will encourage parents to give up on neighborhood schools rather than invest in them.

Rhiannon Wenning, a teacher at Jefferson Junior-Senior High School, said the link between charter schools and open enrollment systems makes her distrustful, even as many of her students are using the choice process to stay at the school after rising home prices pushed them into other parts of the metro area.

“I understand parents want what is best for their child, but part of that as a citizen and a community member is to make your neighborhood school the school that you want it to be,” she said, calling the universal enrollment system an attack on public schools.

Joel Newton of the Edgewater Collective, which provides community support for lower-income schools in the eastern part of the district, said Enroll Jeffco will give the district much better data on which to base decisions, but he worries that Title I schools, which serve large numbers of students from low-income families, won’t be able to compete.

“With an online system like this, it really needs to be a level playing field,” he said. “And in my area, I’d much rather have resources going to curriculum and instructional aides to catch kids up than going into marketing support. But other areas can do that and they have these big, well-funded PTAs.”

Until now, parents have had to seek out information on each school’s website. The online portal starts by asking parents to enter their address and the grade in which they’re enrolling a student. It then displays the parents’ neighborhood school, with an option to explore alternatives. Each school page has extensive information, including a short narrative, descriptions of special programs like math, arts, or expeditionary learning, the school mascot, and the racial and economic breakdown of the student population. The intent, district spokesperson Diana Wilson said, is to let schools “tell their own story.”

Parents can select as many schools as they want when enrollment opens Jan. 22, and they’ll learn in mid- to late February where they got in. However, they have to commit within five days to one school, ending a practice by which parents in the know kept their options open through the summer months. District officials say this will help them plan and budget better.

Kristen Harkness, assistant director for special education in Jeffco, served on the steering committee that developed the system, and she’s also a parent in the district. Even as a district employee who thought she knew the process inside and out, she managed to miss a deadline for her son to be considered at another middle school.

She said that choosing between schools isn’t a matter of which schools are better but which are a better fit for a particular student. In her case, her son could have stayed at a K-8 or transferred to a combined middle and high school, with each option presenting a different kind of middle school experience. He’s happy at the K-8 where he stayed, she said, but parents and students should have the chance to make those decisions.

The new universal enrollment system is poised to give more families that chance. In the course of the rollout, though, there may be a few glitches.

“We’re doing all we can to look into the future and foresee any technical problems and design solutions to that proactively,” Glass said. “That said, this is our first time, and we ask for people’s patience.”