Behind the numbers

Why ‘personalized learning’ advocates like Mark Zuckerberg keep citing a 1984 study — and why it might not say much about schools today

PHOTO: TechCrunch/Creative Commons
Facebook founder Mark Zuckerberg.

Facebook founder Mark Zuckerberg made a bold statement in a recent essay: By giving students individual help, average students can be turned into exceptional ones.

“If a student is at the 50th percentile in their class and they receive effective one-on-one tutoring, they jump on average to the 98th percentile,” Zuckerberg wrote.

It’s a remarkable claim, one that strains the limits of belief. And for good reason: The results from the 1984 study underlying it have essentially never been seen in modern research on public schools.

Still, the results have become a popular talking point among those promoting the “personalized learning” approach that Zuckerberg’s philanthropy is advancing. One video created by the Chan Zuckerberg Initiative features an illustration of a 50 on a graph zooming upward to hit 98. The New Schools Venture Fund, another influential education group that backs personalized learning, cites the same work by Benjamin Bloom.

But a close look at the study raises questions about its relevance to modern education debates and the ability of new buzzed-about programs to achieve remotely similar results.

“If you’re really going to make these huge investments and huge pushes [based on this study], you might want to be absolutely sure that the analysis of that research is solid,” said Ben Riley, head of the group Deans for Impact and a skeptic of personalized learning.

Jim Shelton, who heads CZI’s education work, said in an interview that the organization relies on a great deal of other research, but highlights Bloom to illustrate in the best case scenario for what schools might accomplish.

“It stands to reason that many kids that currently perform at levels that we consider average or even below average could be performing at levels that we would consider superlative,” he said.

Questions then and now about the meaning of Bloom’s work

The conclusions on the effects of tutoring from Bloom’s widely-cited paper are drawn from two studies conducted by University of Chicago graduate students.

One of those studies is available online, but reading the other requires some sleuthing. (We ended up paying for access through a service that compiles dissertations.)

In both studies, students were taught novel subject matter — probability or cartography — using different methods over the course of a few weeks. Some students were taught in a traditional lecture style, others received “mastery-based” teaching, and others received small group tutoring.

On a final test, students who were tutored one-on-one or in small groups came out far ahead, and in some cases the average tutored student beat 98 percent of those taught in the traditional way. Students who received the mastery-based teaching — which overlaps with modern conceptions of personalized learning — also did much better, though not as well as those tutored.

Jim Shelton of the Chan Zuckerberg Initiative in one of the organization’s video, saying that the average student will move to 98th percentile with one-on-one tutoring.

The applicability of these studies today is an open question. Combined, the studies focus on just three schools and a few hundred students. And since this was done more than 30 years ago, things like what traditional instruction looks like may have substantially changed.

The papers include little information about those final tests, but it appears they were designed by the researchers, unlike a traditional standardized test. Researcher-created assessments on subjects that are totally new to students — like cartography and probability, in this case — tend to see students make the largest gains.

Bloom’s work also doesn’t focus on technology-based tutoring, a point personalized learning advocates usually acknowledge. “If it supports anything, it supports one-on-one human tutoring,” Riley said.

But what earned the most attention, then and now, is how big of an impact tutoring had on students. The difference between tutoring and traditional instruction after just three weeks was two standard deviations — to researchers, a truly incredible result. It means bringing students from average to exceptional.

“I’ve never seen a study in education that found effects in the range of two standard deviations, so it’s remarkable for that reason,” said Jon Guryan, a Northwestern professor who has done research on tutoring.

Another researcher, Robert Slavin of Johns Hopkins University, logged concerns about Bloom’s outsize claims as early as 1987. Focusing on such unusually large gains, he wrote, “is misleading out of context and potentially damaging to educational research,” since it could lead researchers to “belittle” more realistic results.

Guryan’s recent work, on tutoring of struggling students in Chicago, found what would normally be considered fairly large gains: about a quarter of a standard deviation on math standardized tests. Other recent research on intensive tutoring in public schools looks similar, in some cases showing even smaller effects. Meanwhile, studies on computer-based personalized learning have shown a range of effects — but none comes close to two standard deviations.

Bror Saxberg, CZI’s vice president of learning science, acknowledged that Bloom’s findings are bigger than in other research. But he said human and computer tutoring can have a substantial impact, pointing to a 2011 overview of research where results come close to a full standard deviation. (This overview included studies in a variety of contexts, including outside K-12 education.)

In sum, a number of studies suggest that Bloom’s huge results are not plausible to expect in public schools today, and they have rarely been seen in other research. Meanwhile, Zuckerberg, Shelton, and CZI’s public statements imply that, with the right tools, students could see similar off-the-charts improvements.

Can ‘personalized learning’ drive huge gains? Advocates hope so.

Shelton analogized Bloom’s work to the human quest to run a four-minute mile: a crazy-seeming goal that was eventually attained by a small number of elite runners.

“Everyone said it was impossible to break the four-minute mile, until somebody broke the four-minute mile,” Shelton said. “Someone has broken the four-minute and its equivalent and we need to figure out how to do it and how to get a lot more people to be able to do it.”

Many others also see Bloom’s research less as a precise accounting of the results of tutoring and more as a call to action. Indeed, most of Bloom’s paper amounts to him pondering a question philanthropists are grappling with today: How can schools get the benefits of individual tutoring without the prohibitive expense of actually hiring each student their own tutor?

“If the takeaway from Bloom is that by doing tutoring and mastery you’re going to get two [standard deviation] gains — I don’t think that’s the right takeaway,” said Todd Rose, a Harvard professor who has argued that schools need greater customization. (CZI has funded some of Rose’s work.)

The value of the study, he says, is that “it speaks to a very different view of human potential than is embedded in our current system.”

Debbie Veney, a spokesperson for New Schools Venture Fund, which is supported by CZI, had a similar take: “[Bloom’s results] inspired and challenged many to figure how to achieve similar conditions in a more cost-effective way — which spawned many creative concepts and efforts to scale similar results.”

That’s in line with CZI’s sweeping ambitions — “empower every teacher everywhere,” as described in one CZI video — and deep pockets.

Zuckerberg and his wife Priscilla Chan have pledged to donate 99 percent of their Facebook shares — worth an estimated $45 billion in late 2015 — to CZI over their lifetime. The organization — which also focuses on criminal justice, immigration, and economic policy — is expected to give “hundreds of millions of dollars” per year to education causes.

The group has already supported a number of tech-based approaches to school, including the Summit learning platform, a computer program created by a charter network to help teachers personalize learning. CZI has also tried to broaden the definition of personalized learning, funding organizations that offer free eye exams and small-group, in-person tutoring.

A spokesperson pointed to other research CZI relies on, including psychological studies from Rose and others on how children learn and develop and the work of Stanford professor Carol Dweck, which suggests that people with a “growth mindset” are more likely to succeed.

But Sarah Reckhow, who studies education philanthropy at Michigan State University, suggests that CZI’s ambitious goals will meet the hard realities of the classroom and fall far short of Bloom’s results.

“I do think they’re setting themselves up to fail,” she said. “If you look at educational research, if you look at what will most definitely vary once you to put something into practice … those effect sizes won’t be replicated, but also there will probably be some cases where it will not turn out well or there will be unintended consequences.”

Asked about his benchmarks for success, Shelton said it’s not clear yet what is possible.

“We’re at the beginning of our journey, not the end of our journey,” he said. “We are in the business of trying to figure out how to solve this problem that has never been solved before.”

another path

‘They’re my second family.’ Largest Pathways to Graduation class earn their diplomas

Jasmine Byrd receives an award for excellence after giving a speech to her fellow graduates.

Before last fall, Jasmine Byrd never envisioned herself striding across the stage to receive a diploma at a graduation ceremony.

But then Byrd moved to the Bronx from Utah and entered New York City’s Pathways to Graduation program, which helps 17- to 21-year-olds who didn’t graduate from a traditional high school earn a High School Equivalency Diploma by giving them free resources and support.

Just walking into this space and being like, this is what you’ve accomplished and this is what you’ve worked hard for is a great feeling,” said Byrd, who also credits the program with helping her snag a web development internship. “I’ve built my New York experience with this program. They’re my second family, sometimes my first when I needed anything.”

Byrd is one of about 1,700 students to graduate during the 2017-2018 school year from Pathways, the program’s largest graduating class to date, according to officials.  

This year, students from 102 countries and 41 states graduated from Pathways, which is part of District 79, the education department district overseeing programs for older students who have had interrupted schooling.

The program also saw the most students ever participate in its graduation ceremony, a joyful celebration held this year at the Bronx United Palace Theater. According to Robert Evans, a math teacher at one of the program’s five boroughwide sites and emcee of the graduation, about 600 students typically show up to walk the stage. But students can be a part of the ceremony even if they received their passing test results that morning, and this year more than 800 graduates attended.

There were still students coming in last night to take photos and to pick up their sashes and gowns,” said Evans.

The graduation ceremony is unique in part because the program is. Students who have not completed high school attend classes to prepare to take the high school equivalency exam. But the program also prepares students to apply for college, attend vocational school, or enter the workforce by providing help applying for colleges, creating resumes and other coaching.

To make sure that the program is accessible to all students, there’s a main site in every borough and 92 satellite sites, located in community centers and youth homeless shelters like Covenant House. Students who want to work in the medical field, like Genesis Rocio Rodriguez, can take their courses in hospitals. Rodriguez, who graduated in December, is now enrolled in the Borough of Manhattan Community College, and passing the exam meant being one step closer to her dream of becoming a nurse.

When I got my results I was with my classmate, and to be honest I thought I failed because I was so nervous during it. But then I went online, and I was like, ‘Oh my gosh I did it!’ My mom started crying and everything.”

Byrd said the program worked for her because of the supportive teachers and extra resources.

“The teachers are relatable,” said Byrd. “They don’t put on an act, they don’t try to separate the person from the teacher. They really reach out, even call you to get you out of bed in the morning.”

Carmine Guirland said the supportive environment of social workers, guidance counselors, and teachers is what attracts him to the work at Bronx NeOn, a site where students who are on probation or who are involved with the court system can prepare for the exam, college, and careers.

When students are on parole they will have really involved [parole officers] who would text me at the beginning of class to check in so that we could work together,” said Guirland. “It’s really about that village thing. The more support systems that are available the more success the students will have.”

Reflecting on his experiences with the graduating class, Guirland’s most treasured memory was when one of his students proposed to his girlfriend in a guidance counseling session. Even though they aren’t together anymore, the moment was a reflection of the relationships that many of the students build during their time at Pathways to Graduation.

“It’s this amazing high moment where this student felt like the most comfortable place for him to propose to his girlfriend and the mother of his child was in our advisory circle,” said Guirland.

New Standards

Tennessee updates science standards for first time in 10 years. New guidelines stress class discussion, inquiry

PHOTO: Rebecca Griesbach
Fourth grade science teachers Lamarcus Marks, of Rivercrest Elementary, and Angie Clement, of Bartlett Elementary, test out a lesson on kinetic and potential energy at Arlington High School, one of 11 statewide sites where Tennessee teachers are training for next year's new science standards.

How can a wolf change the river? Why doesn’t a cactus have leaves? Why can’t you exterminate bats in Tennessee?

With new state science standards coming to classrooms next fall, these are the kinds of questions students will explore in their science classes. They’ll be tasked not only with memorizing the answers, but also with asking questions of their own, engaging on the topic with their teacher and classmates, and applying what they learn across disciplines. That’s because the changes set forth are as much about teaching process, as they are about teaching content.

“At the lowest level, I could just teach you facts,” said Detra Clark, who is one of about 300 Tennessee educators leading teacher trainings on the new standards to her peers from across the state. “Now it’s like, ‘I want you to figure out why or how you can use the facts to figure out a problem.’”

PHOTO: Rebecca Griesbach
Detra Clark, a science coach in Shelby County’s iZone, demonstrates a sample lesson for sixth grade science teachers.

On Wednesday, Clark — a science coach for the iZone, a group of underperforming schools that Shelby County Schools is looking to turn around — unpacked for her peers, who gathered at Arlington High School, a key component of the new material: three-dimensional modeling. Under three-dimensional modeling, students should be able to do something with the content they learn, not just memorize it.

In recent years, Tennessee students have performed better on state science tests than on their math and English exams. But state science standards for grades K–12 haven’t been updated since 2008. By contrast, math and English benchmarks have undergone more recent changes. To give the stakeholders time to adjust, results from next year’s science test, the first to incorporate the new standards, won’t count for students, teachers, or schools.

At the training session, Clark, standing before a room of sixth-grade science teachers, held up a chart with the names of woodland animals, such as elk and deer. Under each name, she tracked the population over time.

“At our starting population, what do we see?” she asked.

“The deer, it decreases again because it’s introduced to a predator,” a teacher responded.

“More resources, more surviving animals” another teacher chimed in.

“How can we explain what happened in year two, when we’re dealing with students?” Clark asked the group.

“The population went up,” a teacher said.

“They start to reproduce!” another teacher interjected.

Clark nodded.

In another classroom, this one composed of kindergarten teachers, Bridget Davis — a K-2 instructional advisor for Shelby County Schools — clicked through a video of fuzzy critters, each paired with a close relative, such as two different breeds of dogs.

PHOTO: Rebecca Griesbach
In a teacher training session on Wednesday, kindergarten teachers highlight the three dimensions of three-dimensional modeling, a key part of new state science standards.

She encouraged the teachers to ask their students what traits the animals shared.

“The first thing they’re going to say is, ‘Well, one’s big and one’s small,” she said. “What we really want them to say is, ‘Well, their fur is the same color,’ or, ‘Mom has a patch of black hair here and the baby doesn’t.’ We want them to look at detail.”

She added, “We want them to get used to being a detective.”

The science standards that have been in place for the past decade fulfills the first dimension of three-dimensional modeling.

Doing something with that knowledge satisfies the second dimension, and the third dimension requires teachers to apply to their lessons a “cross-cutting concept” — strategies that students can apply to any subject, like identifying patterns or sequences.

Under the existing standards, a student may not have been introduced to physical science until the third grade. But starting next year, Tennessee schoolchildren will learn about life science, physical science, earth and space science, and engineering applications, beginning in kindergarten and continuing through high school.

“I do believe that this is the best our standards have ever been, because of the fact that they are so much more detailed than they have been in the past,” Davis said.

About a thousand Shelby County teachers made their way to trainings this week, which were free and open to all educators. Several administrators also met to discuss ways they can ensure the new standards are implemented in their schools.

As with anything new, Jay Jennings — an assistant principal at a Tipton county middle school and an instructor at Wednesday’s training — expects some pushback. But he’s optimistic that his district will have every teacher at benchmark by the end of the 2018–2019 school year.

“We talked before about teachers knowing content, and that’s important,” he said. “But what we want to see is kids knowing content and questioning content. We want to see them involved.”

He reminded other school leaders about last year’s changes to English and math standards, a transition that he said was challenging but smoother than expected.  

“Teachers are going to go out of their comfort zone,” he explained. “But it’s not changing what a lot of them are already doing.”