A critical question has hung over the release of scores on national math and reading tests: Can state trends be relied on, given this year’s switch to digital tests?
For the first time, the vast majority of students took the National Assessment of Educational Progress on tablets in 2017. Students can be affected by how they take a test, something researchers call “mode effect” — and NCES, the federal agency that administers the tests, says it’s gone to great lengths to ensure that comparisons over time are fair.
But NCES has found itself having to defend its own methods publicly, as a state education chief raised questions about the comparisons and even asked outside researchers to analyze the data before its official release.
Louisiana superintendent John White registered concerns regarding NAEP’s digital switch and what that meant for students with less familiarity with digital assessments in a March 23 letter. White didn’t mention his own state’s results then, but the scores released Monday show that the state saw approximately 5-point drops on both fourth grade exams, a 2-point decline in eighth grade math, and a 1-point gain in eighth grade reading.
In a letter obtained by Chalkbeat, NCES associate commissioner Peggy Carr responded to White. She wrote that students nationally did fare worse on digital tests compared to paper, with fourth-graders seeing scores dip more than eighth graders. Those differences — the “digital penalty” — were accounted for with a national adjustment when NCES reported the latest results, Carr notes.
After performing those adjustments, Carr wrote, “For Louisiana … we found no statistically significant differences between the [digital] and [paper] results.” In essence, NCES looked at whether that digital penalty was worse in Louisiana — based on its students who took the paper tests — than in the country as a whole, and finds little evidence of that.
White points out that there were some modest differences in his state, even if they weren’t statistically significant. If the results were based on the the paper exam, the state’s fourth-grade reading scores would only have declined by 1.6 points, rather than 4.6; in eighth grade math, a nearly 2-point decline would have been a half-point gain.
“There is certainly some caution with which we need to view results,” White told Chalkbeat. “My point is you’ve got variation.”
Yet in fourth grade math and eighth grade reading, Louisiana’s digital penalty was actually modestly smaller than the country as a whole, going against White’s concern that his state was particularly affected
The extent to which different states had varying digital penalties was also minimal, inconsistent, and usually not statistically significant, Carr told reporters ahead of the scores’ release.
But an analysis conducted by Johns Hopkins University’s Institute for Education Policy found that states where students had prior experience taking state exams online saw slightly larger gains — at most, 2 points — on three of the four tests.
The results only show that having taken online tests is correlated with those gains; it doesn’t prove that explanation.
Carr declined to comment on the Johns Hopkins report since it had been conducted before the NAEP scores were formally released. White said that he had shared results with the Hopkins researchers, with the stipulation that they honor NCES’s release timeline.
Read Carr’s response in full: