Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

new money

House budget draft sends more money to schools, but not specifically to teacher raises

PHOTO: Laura Faith Kebede/Chalkbeat

Despite months of heated debate, Indiana House Republicans are not setting aside extra dollars for meaningful teacher raises in their version of the state’s $14.5 billion education budget plan released Monday night.

Even though lawmakers are proposing preserving a controversial merit-based bonus pool and adding small amounts for teacher training programs, their budget draft would largely leave it up to school districts to dole out raises through increased overall funding.

The budget draft proposes increasing what Indiana spends on schools overall by $461 million — or 4.3 percent — through 2021, a little more than increases in years past. The basic per-student funding that all districts get would jump from $5,352 per student this year to $5,442 per student in 2020 and $5,549 per student in 2021. House lawmakers are also adding in a one-time payment of $150 million from state reserves that would pay down a pension liability for schools. But while lawmakers and Gov. Eric Holcomb have said that pension payment would free up about $70 million in schools’ budgets each year, the state likely wouldn’t require the cost-savings be passed along to teachers.

Although increasing teacher pay is a top goal for House Republicans, lawmakers have crafted bills that hinge on districts spending less money in areas such as administration or transportation rather than adding more money to school budgets and earmarking it for teacher salaries.

Their criticism of school spending has raised the ire of superintendents and educators who say they have little left to cut after years of increasing costs and state revenue that has barely kept pace with inflation.

But budget draft, which is expected to be presented to and voted on by the House Ways and Means Committee on Tuesday, doesn’t completely omit efforts to incentivize teachers to stick around. Unlike Holcomb’s budget proposal, House lawmakers are keeping in the current appropriation of $30 million per year for teacher bonuses.

The House budget draft would also set aside $1 million per year for a teacher residency pilot program and $5 million per year for schools that put in place career ladder programs that allow teachers to gain skills and opportunities without leaving the classroom.

Teacher advocacy groups, such as the Indiana State Teachers Association and Teach Plus, have been supportive of residency and career ladder programs, but the organizations have also called for more action this year to get dollars to teachers. Additionally, the ideas aren’t new — similar programs have been proposed in years past.

Calls for the hundreds of millions of dollars it would take to raise teacher salaries to be more in line with surrounding states will likely go unheeded for now as the state instead prioritizes other high-profile and expensive agencies, such as the Department of Child Services and Medicaid.

But while plans for major teacher pay raises appear to be on hold, House lawmakers are looking to boost funding in other areas of education to support some of the state’s most vulnerable students.

The budget draft would increase what the state must spend on preschool programs for students with disabilities from the current $2,750 per-student to $2,875 in 2020 and $3,000 in 2021 — the first such increase in more than 25 years.

House lawmakers are also proposing the state spend more money on students learning English as a new language, at $325 per student up from $300 per student now. While all schools with English learners would receive more money per student under this plan, the new budget draft removes a provision that had previously allocated extra dollars to schools with higher concentrations of English learners.

A 2017 calculation error and an uptick in interested schools meant state lawmakers did not budget enough money for schools with larger shares of English-learners in the last budget cycle, so they ended up getting far less than what the state had promised. But even the small increases were valuable, educators told Chalkbeat.

House lawmakers also suggested slashing funding for virtual programs run by traditional public school districts. Going forward, funding for both virtual charter schools and virtual schools within school districts would come in at 90 percent of what traditional schools receive from the state — now, only virtual charter schools are at the 90 percent level. It’s a marked change for House lawmakers, who in years past have asked that virtual charter school funding be increased to 100 percent.

The virtual funding proposal comes as lawmakers are considering bills that would add regulations for the troubled schools, where few students pass state exams or graduate.

The budget draft also includes:

  • $5 million per year added to school safety grants, totaling $19 million in 2020 and $24 million in 2021
  • Doubling grants for high-performing charter schools from $500 per student to $1,000 per student, at a cost of about $32 million over two years. The money is a way for charter schools to make up for not receiving local property tax dollars like district schools, lawmakers say.
  • $4 million per year more to expand the state’s private school voucher program to increase funding for certain families above the poverty line. Under the plan, a family of four making between $46,000 and $58,000 annually could receive a voucher for 70 percent of what public schools would have received in state funding for the student. Currently, those families receive a 50 percent voucher.
  • About $33 million over two years (up from about $25 million) for the state’s Tax Credit Scholarship program.

rethinking the reprieve

Indiana lawmakers take step to eliminate generous ‘growth-only’ grades for all schools, not just those in IPS

PHOTO: Anthony Lanzilote

A panel of Indiana lawmakers took a first step Monday to stop giving new and overhauled schools more generous state A-F grades that consider only how much students improve on tests and cut schools slack for low test scores.

The House Education Committee was initially looking to clamp down on Indianapolis Public Schools’ innovation schools, barring them from using student test score improvement as the sole determinant in their first three years of A-F grades. The more generous scale has boosted IPS’ performance as it launches a new strategy of partnering with charter operators, by allowing some innovation network schools to earn high marks despite overall low test scores.

But lawmakers expanded the scope of the bill to stop all schools from receiving what are known as “growth-only grades” after Chalkbeat reported that IPS’ overhauled high schools were granted a fresh start from the state — a move that would allow the high schools to tap into the more lenient grading system.

“I want to be consistent, and I felt like [grading] wasn’t consistent before, it was just hodge-podge,” said committee Chairman Bob Behning, an Indianapolis Republican. “We need to be transparent with parents.”

Read: Why it’s hard to compare Indianapolis schools under the A-F grading system

The committee unanimously approved the bill. If it passes into law, Indianapolis Public Schools stands to be one of the districts most affected. Growth-only grades for innovation schools have given the district’s data a boost, accounting for eight of the district’s 11 A grades in 2018. All of its high schools could also be eligible for growth-only grades this year.

Indianapolis Public Schools officials did not immediately respond to requests for comment. In the past, they have defended the two-tiered grading system, arguing that growth on state tests is an important window into how schools are educating students. Growth-only grades were originally intended to offer new schools time to get up and running before being judged on student test scores.

IPS was also the target of another provision in the updated bill that would add in stricter rules for when and how schools can ask for a “baseline reset” — the fresh start that its four high schools were recently granted.

Read: IPS overhauled high schools. Now, the state is giving them a fresh start on A-F

The resets, which districts can currently request from the state education department if they meet certain criteria that show they’ve undergone dramatic changes, wipe out previous test scores and other student performance data to give schools a fresh start. The reset schools are considered new schools with new state ID numbers.

The state determined a reset was necessary for IPS’ four remaining high schools because of the effects of decisions last year to close three campuses, shuffle staff, and create a new system a new system for students to choose their schools. Each school will start over with state letter grades in 2019.

But Behning and other lawmakers were skeptical that such changes merited starting over with accountability, and they were concerned that the process could occur without state board of education scrutiny. If passed into law, the bill would require the state board to approve future requests for accountability resets.

A state board staff member testified in favor of the change. The state education department did not offer comments to the committee.

Rep. Vernon Smith, a Democrat from Gary, said he didn’t like the fact that a reset could erase a school’s data, adding that he had concerns about “the transparency of a school corporation getting a new number.”

The amended bill wouldn’t remove the reset for IPS high schools, but by eliminating the growth-only grades, it would get rid of some of the incentive for districts to ask for a reset to begin with. Under current law, reset schools are considered new and qualify for growth-only grades. But the bill would require that reset schools be judged on the state’s usual scale, taking into account both test scores and test score improvement — and possibly leading to lower-than-anticipated state grades.

The amended bill would still offer a grading grace period to schools opening for the first time: New charter schools would be able to ask the state to give them no grade — known as a “null” grade — for their first three years, but schools’ test score performance and test score growth data would still be published online. Behning said he didn’t include district schools in the null-grade measure because they haven’t frequently opened new schools, but he said he’d be open to an amendment.

The bill next heads to the full House for a vote.