It was a small detail tucked inside a March 2014 New York Times Magazine piece titled “The Story Behind the SAT Overhaul.”
The piece profiled the College Board’s new leader, David Coleman, and a suite of initiatives he was introducing to make the test — and the entire college admissions process — fairer. That included sending packets of information to low-income, high-achieving students about college, something that had been shown to push them into more selective universities.
“Research on the initial effects of the program won’t be released until next month,” the story said.
But that research didn’t come out the following month, or the month after that. The results, showing that the packets made virtually no difference after all, weren’t published until five years later.
“This was the pattern with the College Board during the early years of David Coleman’s presidency: one grand, well-publicized attempt after another to make the SAT more equitable and more fair ending in either quiet unpublicized failure or in a noisy claim of success that fell apart under more careful scrutiny,” argues journalist Paul Tough in his new book, “The Years that Matter Most.”
The book catalogs instances where the College Board spun, obscured, or downplayed unfavorable research findings. In Tough’s telling, the initiatives — including free SAT prep created by Khan Academy — amounted to a “large-scale exercise in corporate rebranding” for the nonprofit that had been losing market share to competitor ACT, Inc.
The College Board, and Coleman himself, deny that. “There’s no mercantile motive,” Coleman told Chalkbeat. “We do our best to make tools available for students from a wide range of backgrounds to show their talents.”
In his telling, the College Board made earnest efforts to address — and study — vexing societal inequities. The results may have been mixed, even concerning in some cases, but that reflects the difficulty of the mission, not the failure of the College Board, he said. The College Board released a lengthy response to Tough’s book.
Tough’s portrayal has the potential to shift the perception of one of the most influential players in education as it continues to roll out high-profile initiatives purporting to reduce inequity in higher education. And Tough’s indictment is broader than the College Board — it includes philanthropists and policymakers eager to find cheap solutions to entrenched disparities and news outlets that accepted the College Board’s framing.
Curious about Tough’s claims? We took a careful look ourselves. Here’s what we know about the organization’s key initiatives and how the College Board is responding.
What we know: A packet of college info sent to low-income high achievers didn’t help them much — and the College Board sat on the results for years, though it did eventually release them.
The College Board’s packets were based on work by researchers Caroline Hoxby and Sarah Turner. They found that providing customized information about the college application process, as well as application fee waivers, caused those high achievers to attend colleges better matched with their abilities.
A New York Times columnist called it “a simple way to send poor kids to top colleges.” Coleman decided to take the effort nationwide shortly after taking the helm at the College Board.
That’s where Tough picks up the trail. He says he asked for results of that expansion, only to be put off again and again. Tough finally received an internal memo about the program’s first two rounds. “As soon as I read it, I understood why the College Board had been so reluctant to give it to me,” Tough wrote. The effects were minimal.
As Tough’s book was going to press, a study was released looking at the program’s effect on a larger group of students over the following two years. Again, the packets had few clear effects. (Chalkbeat wrote about the study after finding it on a research website two weeks after its release; the College Board did not publicize the findings.)
Today, the College Board says it’s stopped sending the packets but incorporated elements of them into other projects.
While the results were disappointing, they can be seen as the social scientific process in action: trying something that other research suggested would be successful, monitoring the results, and discontinuing what doesn’t work.
Tough counters that the new study got too little attention, too late. “They knew these results years ago and they chose … not to reveal to the public that this replication wasn’t working,” he said in an interview.
“We were willing to share those results,” Coleman told Chalkbeat, pointing to the 2019 study. “But admittedly, we made some false starts where we thought we’d earlier be able to share the results, and we paused because we found this whole area rather confusing, what we were seeing.”
Coleman said that the organization was also concerned about seeming to differ with researchers the College Board respected.
“I totally accept that another road we could have gone down, in retrospect, was to give gradual reports of progress,” he said. “I think that might have been better, and I just want to own that.”
What we know: There’s no evidence that the College Board’s partnership with Khan Academy has helped close racial or economic test-score gaps.
The College Board partnered with Khan Academy to provide free online SAT prep courses starting in 2015. Its goal was to provide a credible alternative to the expensive tutoring services accessible to affluent students.
One year in, Coleman claimed remarkable results. “Never in my career have I seen a launch of technology on this scale that has broken down the racial divisions that so haunt this nation — never,” Coleman said.
Two years in, the College Board released data showing that time spent on Khan Academy test prep was correlated with similar gains for students of all demographics. Practice, Coleman said, is “an equal opportunity employer.”
The problem, Tough notes, is that the data showed that not all groups of students were practicing the same amount. Those with more educated parents tended to study longer, and Asian students used the tool far more than any other racial group. (Black students used it a bit more than white and Hispanic students.)
Coleman says those disparities concern the College Board, too, and that the organization has acknowledged this in its research reports. “The notion that a free tool would disrupt those patterns on day one would be unbelievable, really,” he said.
The College Board says it’s taken meaningful action to respond to the data, such as partnerships with urban school districts and incentives for low-income students to use Khan Academy.
The College Board also points out that students who had the lowest PSAT test scores saw the biggest gains after more time on Khan Academy. But again, students with higher scores were much more likely to spend a lot of time practicing.
Another key point to understanding this debate, which Tough doesn’t get into, is that the College Board’s research doesn’t show whether Khan Academy truly caused the score increases. Perhaps the students who used Khan most were particularly motivated or were using other study aids.
The bottom line: Students’ use of Khan Academy was unequal. It’s also possible that Khan partially replaced an even more unequal system of private tutoring and commercial prep programs, and has prepared students who might otherwise have gone without any extra help. It’s impossible to tell what the overall effect has been, but Coleman’s claims from 2016 seem overstated.
What we know: The SAT favors affluent students more than grade point averages do. But there’s also little evidence that more colleges going “test optional” would level the playing field.
Tough also takes aim at an analysis of high school grade inflation published in a 2018 book edited by College Board-affiliated researchers. It shows that grades at high schools serving affluent and white students are rising faster than GPAs at schools serving mostly students from low-income families. Separate research in North Carolina has found a similar phenomenon.
These results were widely publicized, and the College Board used them to suggest the downside of moving away from admissions tests like the SAT.
“Submitting SAT scores as part of a college application can open doors to opportunity not just for a privileged few, but for all students,” reads a College Board-sponsored piece focused on the study’s findings.
But Tough points out that other aspects of the study, as well as other research, don’t support the idea that GPA gives more advantaged students a leg up.
At the student level, GPA increases between 2001 and 2016 were similar for black students (+.11) and white students (+.12). Students with highly educated parents saw only slightly larger increases (+.15) than students whose parents hadn’t completed high school (+.11). (One exception was Hispanic students, who have seen minimal GPA increases over that time.)
SAT scores are correlated with race and family income, much more so than high school GPA — a fact other researchers note in the same book. “Studies have found that students who have higher [high school] GPAs than test scores tend to be female, minority, and low-income students,” they write.
Coleman says the key fact is that the SAT can help predict, along with GPA, which students will do well in college.
“That debate is to me, tired,” he said. “What’s evident is that two sources of data, like so many cases in life, are more than one.” (Research supports that claim, though high school GPA alone is more predictive than SAT scores alone.) Coleman has also said that poor SAT scores “must not be a veto on any student’s life.”
Tough zeroes in on expensive test prep as a key driver of the disparities between the SAT scores of different groups. But here he’s on weaker ground: research has found that test prep offers students only a modest boost, and large test score gaps exist on exams that students are unlikely to explicitly prepare for.
What all of this means for the fight to make college admissions more equitable is unclear. Tough is highly critical of the SAT, but acknowledges that empirical research has not found that colleges become more racially or economically diverse after they drop SAT or ACT requirements.
Coleman also argues that, if more schools shifted attention from test scores to grades, affluent students’ GPAs would rise at an even more rapid clip. “Let’s not be naïve,” he said.
In general, he says, people should not expect that the SAT to mitigate the realities of poverty. As inequality persists, tests will reflect them.
“We can empower more people to practice, to grow, to defy those,” he said. “It should never blind us to their effects.”