Reading between the lines on test score reporting

test books by menlophoto

From the Washington Post, a glaring example of why it’s so important for educators, parents, and concerned citizens to turn a critical eye on education reporting, especially reporting about test scores:

Today, the paper ran a story about across-the-board improvements in test scores in Maryland’s Anne Arundel County, where reading test scores increased by 4 percentage points, to 86 percent proficient, and math test scores increased by 3 percentage points, to 84 percent proficient. Sounds pretty good, right? School leaders attribute the gains to the fact that they broke down test score data to focus on the particular skills individual students needed to improve and to teachers’ increased cultural sensitivity to how children learn. The Post reporter takes her sources at their words, writing, “Countywide results reflect that effort.”

But there is more to this story, and much of it appeared in the Post last week, in an article about statewide test results and the skepticism with which they have been received. First, Anne Arundel County’s scores may have increased, but they didn’t keep pace with the average gain in Maryland — statewide, reading scores increased by six percentage points, and math scores by four percentage points.

In addition, the Post reported just last weekend that Maryland test was shorter this year than in the past and that the state dropped its practice of drawing some test questions from a national exam. That important information didn’t make it into today’s article. A shorter test could have reduced fatigue that might have inhibited students from performing at their best in previous years. And because the test contained fewer items, the results should be considered less reliable than in past years. Finally, by including only questions devised by Maryland teachers in line with the state’s standards, test makers increased the likelihood that students would be able to answer more questions. Together, these changes could account for much, or even all, of the rise in test scores, but not surprisingly, state education officials are uninterested in examining whether that’s true.

Students in Anne Arundel County — home of the elementary school profiled in Tested, Linda Perlstein’s expert 2007 book about the effects of high-stakes testing on students, classrooms, and schools — might be doing better than they ever have before. But the county’s test scores don’t tell us that, and neither does the Washington Post.