I’ll admit it: When I hear the phrase “charter school miracle,” my antennae go up. It’s not that I think that charter schools can’t possibly be good schools, or that they cannot surpass traditional public schools in the measured achievements of their students. The evidence is pretty clear that there are many fine charter schools, just as there are many struggling charter schools.

No, it’s that I think miracles are exceedingly rare phenomena. And the current narrative about miracles in school reform relies heavily on a “great man” theory, replete with outsized personalities. Witness the contemporary stage, on the cusp of the release of Waiting for “Superman”: Geoffrey Canada, Michelle Rhee, even — God help us — Bill Gates and Joel Klein being anointed as miracle-workers who, by dint of their commitments, hard work and personalities, are overcoming entrenched bureaucracies and transforming the life-chances of poor and minority children across America’s urban landscape.

It was against this backdrop that I read Caitlin Flanagan’s stirring op-ed that graced the gatefold of Sunday’s New York Daily News. Flanagan, a former prep-school teacher who now writes for The Atlantic and other publications, singles out Mike Piscal, who founded a charter management organization called the Inner City Education Foundation (ICEF) that now operates 15 elementary, middle and high schools in south Los Angeles. Flanagan and Piscal were colleagues, once upon a time, in the English department of the elite Harvard-Westlake School.

Flanagan’s argument goes something like this: the ICEF schools are extraordinarily high-performing; in fact, the elementary schools have eliminated the achievement gap. But the educational bureaucracy is trying to close them down. These fine schools were unable to benefit from the Race to the Top funds they should have received because California’s RttT application was scuttled by a lack of teacher-union support. These teachers unions, therefore, deserve our scorn: they are single-handedly preventing inner-city children from succeeding.

The gaps in logic are breathtaking. It’s not at all obvious that ICEF charter schools would have gotten a cent if California’s RttT application had been funded. As is true of most RttT applications, California’s emphasized developing new standards and assessments, providing high-quality professional development to principals and teachers, expanding the state’s longitudinal data system and improving its lowest-performing public schools.

Basic operational support for existing schools was never the purpose of the competition. Moreover, California lost more points in the RttT judging for its failure to fully implement a longitudinal data system than for not securing the support of a broad group of stakeholders for its plan. (And teachers unions were not the only stakeholders who were lukewarm about the state’s application.)

But I was most intrigued by Flanagan’s claims about how the ICEF schools have closed the achievement gap. Time and time again, such claims have been shown to be exaggerated. We’d all like to believe the story of how great people, working hard, can overcome the powerful forces that structure inequality in American society. Can it be true?

Here’s what Flanagan wrote: “ICEF has done what we are always told is impossible. All five of its elementary schools have eliminated the achievement gap in reading for its African American students. Eliminated it. That fact alone should cause the Department of Education to send a team of researchers to ICEF this afternoon to keep them there until they learn what Mike’s doing.”

How much of this is true? Well, there are five ICEF elementary schools. Beyond that …

flanagan-fig-1I compared the average performance of students in these five schools on the 2010 statewide California Standards Tests (CST) with the average performance of white students in the Los Angeles Unified School District, expressing the gap as a fraction of a standard deviation. (I estimated the 2010 standard deviations as the average of the 2008 and 2009 standard deviations.) Figure 1 shows the data for English Language Arts for grades 2 through 5. Across the Los Angeles Unified School District, black students score about .85 to .90 standard deviations below white students, on average. A gap of this magnitude indicates that roughly 80 percent of all white students score above the typical black student.

If the achievement gap had been eliminated in the five elementary schools, then the columns expressing the gap should have no height – they’d be at zero. Clearly, that’s not the case. On average, students in ICEF elementary schools have made some progress in closing the achievement gap in reading test performance, but that’s driven primarily by the performance of students at a single school, View Park Prep.

Second-graders at View Park Prep are outscoring the typical white student in Los Angeles, and that’s a remarkable accomplishment. But if one wanted to cherry-pick results to make a point, it would be just as easy to point out that the fourth-grade students at Vista are performing 1.1 standard deviations below the typical white student in L.A. (More than 75 percent of the students at Vista are Hispanic or Latino; there aren’t even enough black students for the state to report their performance on the CST, which makes me wonder why Flanagan felt that she could say that “all five” of the ICEF elementary schools had eliminated the achievement gap in reading for African-American students.)

flanagan-fig-2The story is much the same in mathematics. Figure 2 reports the math achievement gaps for the five ICEF elementary schools. As is true in reading, the district-wide black-white achievement gap in mathematics is substantial – on the order of .9 standard deviations. The ICEF elementary schools have made some progress in closing the gap, and that’s commendable.

But no one looking at this figure would conclude that the ICEF elementary schools have come close to eliminating the achievement gap that separates the test scores of African-American and Latino children from white children in Los Angeles. Test scores are, to be sure, a very narrow representation of what children are learning in school, and I would never want to base a judgment about the quality of ICEF schools, or any other schools for that matter, solely on test scores. But Flanagan flew the achievement-gap flag, and her claims don’t hold up under scrutiny.

I like a good story as much as the next guy. But when it comes to swaying opinion on important matters of public policy, we should demand more. Perhaps Caitlin Flanagan has access to other data that provide more support for her claim that all five of the ICEF elementary schools have eliminated the black-white achievement gap in reading. But until she goes beyond a bald, unsupported assertion, she’s got a credibility gap.

This post also appeared on Eye on Education, Pallas’s column at The Hechinger Report.