Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

NEW MOMENT

Tennessee’s struggling state-run district just hired the ‘LeBron James’ of school turnaround work

PHOTO: Yalonda M. James/The Commercial Appeal
Sharon Griffin was the first chief of schools for Shelby County Schools. Starting in May, she will be the next leader of Tennessee's state-run district.

In hiring a Memphis native to save its most vulnerable schools, Tennessee is hedging its bets that she can finally get the job done.

Sharon Griffin’s new job is to fix the state’s struggling Achievement School District and use her experience to strengthen the relationships with local districts across the state.

But can she right the ship and make everyone happy?

“I know through my experience and the relationships I’ve built that we cannot only focus and prioritize our work, but strengthen the relationship [between local districts and the state] so all of our schools can be great places of learning,” Griffin said during a conference call this week.

Tennessee’s achievement district started out as the cornerstone of the state’s strategy to improve low performing schools in 2012. It promised to vault the state’s 5 percent of lowest-achieving schools to the top 25 percent within five years. But the district hasn’t produced large academic gains. It’s struggling to attract students and retain high-quality teachers. And local districts don’t like it because the state moved in and took over schools without input.

But as Tennessee works to make its state the national model of school achievement, naming a revered, longtime home-grown leader as point person for school turnaround is seen by many as a jolt of badly needed energy, and a savvy move in a state education system divided into many factions.

“I think it is a game-changer,” said Rep. Raumesh Akbari, D-Memphis, who has championed legislation to refine the achievement district. “The ASD badly needs a strong leader…. She definitely could be the bridge to bring us over troubled water in Tennessee.”


Read more about what Griffin’s hire means for the school district she is leaving behind. 


Education Commissioner Candice McQueen stressed during the call that Griffin’s appointment does not mean state-run schools will return to local control, even as she acknowledged that the district is at a turning point. It’s now the state’s tool of last resort.

“Whether that is transitioning a school back into the district when it is ready or whether it’s to intervene and move a school into the Achievement School District,” McQueen said. “This particular moment is about a person who can lead all of the state interventions as well as the specificity of the ASD.”

For Bobby White, the founder and CEO of a Memphis charter organization in the achievement district, the appointment signals a new chapter ahead. Griffin will directly oversee the district’s 30 charter schools in her new role.

He has been around for the highs and lows of Tennessee’s six-year experiment in state-run turnaround work.

“It feels like we got LeBron James, you know?” said White, who runs Frayser Community Schools. “It feels like she will have a vision and take us where we have been needing to go.”

PHOTO: Caroline Bauman
Frayser Community Schools CEO Bobby White has seen the highs and lows of the turnaround district.

Part of that vision will be finding new ways for charter schools and local districts to work together. In her roles as assistant commissioner of School Turnaround and chief of the Achievement School District, Griffin will oversee more than just the state-run district. She will have a hand in turnaround efforts across the state, such as a new partnership zone in Chattanooga. In the partnership zone, state and local leaders will work together to create mini-districts that are freed from many local rules.

Griffin stressed earlier this week that building relationships and fostering collaboration are among her top strengths — efforts that the state has failed in as local districts have sparred with state-runs schools over enrollment, facilities, and sharing student contact information.

“We have a level playing field now,” Griffin said. “I want to be clear, it’s not us against them. It’s a chance to learn not only from what ASD has been able to do alongside charter schools, but a chance to learn from each other as we move forward.”

Marcus Robinson, a former Indianapolis charter leader, said Griffin’s dynamic personality will be enough to get the job done.

“Dr. Griffin is magnetic,” said Robinson, who has helped raise money for Memphis schools through the Memphis Education Fund.

“She is the type of person who disarms people because she’s so authentic and genuine,” he said. “But she’s also experienced and wise and she knows school turnaround work.”

Griffin leaves behind a 25-year award-winning career with Shelby County Schools, the local district in Memphis. She has been a teacher and principal. She spearheaded the district’s turnaround work, and now serves as chief of schools. She will start her new role in May and will stay based in Memphis — something community members have long asked for.

Student at Frayser Achievement Academy.
PHOTO: Kyle Kurlick
A student walks through the hall of Frayser Achievement Elementary School, a state-run school.

Steve Lockwood has watched the state’s reform play out in his Memphis neighborhood of Frayser, whose schools were home to some of the first state takeovers.

When the state first started running schools in Frayser, it was with the promise that the academics and culture would improve, said Lockwood, who runs the Frayser Community Development Corporation.

“The ASD has struggled to deliver on their mission,” Lockwood said. “But the last few months have been modestly encouraging. The ASD has seemed willing to admit mistakes and shortcomings.”

Lockwood said he sees Griffin’s appointment as a commitment by the state to bettering relationships in Memphis — and added that he was surprised she signed up.

“It’s a tribute to the ASD that they have enough juice left to attract someone like Dr. Griffin,” Lockwood said.

Senior reporter Marta Aldrich contributed to this report.

unforced error

Mayor de Blasio says education department has culture of frivolous harassment complaints

PHOTO: Patrick Wall
Mayor Bill de Blasio

A “hyper-complaint dynamic” within the city’s education department explains why so few of the harassment claims made against the agency are substantiated, Mayor Bill de Blasio said Wednesday.

“It’s pretty well known inside the education world of some people bringing complaints of one type or another for reasons that may not have to do with the specific issue — and this is not just about sexual harassment,” de Blasio said at a press conference.

“We have to investigate everything but it is a known fact that unfortunately there has been a bit of a hyper-complaint dynamic sometimes for the wrong reasons.”

The mayor’s comments come less than a week after the city released statistics that show nearly 500 education department employees filed sexual harassment complaints over the past four years — but just seven of the complaints were substantiated, according to the New York Times. That means only 2 percent of complaints were found to have merit — compared with nearly 17 percent at other agencies citywide.

During a question and answer session with reporters, de Blasio repeatedly said the education department has a cultural problem when it comes to reporting misconduct.

“I can’t parse out for you who was sincere and who was insincere and what type of offense,” de Blasio said. “I can’t get there. I can tell you the fact it’s unfortunately a part of the culture of an agency that is changing that we need to address.”

De Blasio quickly tried to walk back some of his comments on Twitter.

The mayor’s comments come as activists worldwide have raised awareness about sexual harassment, sparking the #MeToo movement. One element of that conversation has been the  importance of taking harassment claims seriously instead of dismissing them. More than three-quarters of the city’s teachers are women, according to the Independent Budget Office.

De Blasio’s responses drew sharp criticism from Michael Mulgrew, president of the city’s teachers union. “Our teachers have a tough enough job that they don’t have time to make frivolous claims,” he said in a statement.

Schools Chancellor Richard Carranza, who was accused of gender discrimination when he was a top school district official in San Francisco, said the education department has increased the number of investigators who look into such complaints.