testing questions

Did computer testing muddle this year’s NAEP results? Testing group says no; others are unconvinced

PHOTO: Grace Tatter

A critical question has hung over the release of scores on national math and reading tests: Can state trends be relied on, given this year’s switch to digital tests?

For the first time, the vast majority of students took the National Assessment of Educational Progress on tablets in 2017. Students can be affected by how they take a test, something researchers call “mode effect” — and NCES, the federal agency that administers the tests, says it’s gone to great lengths to ensure that comparisons over time are fair.

But NCES has found itself having to defend its own methods publicly, as a state education chief raised questions about the comparisons and even asked outside researchers to analyze the data before its official release.

Louisiana superintendent John White registered concerns regarding NAEP’s digital switch and what that meant for students with less familiarity with digital assessments in a March 23 letter. White didn’t mention his own state’s results then, but the scores released Monday show that the state saw approximately 5-point drops on both fourth grade exams, a 2-point decline in eighth grade math, and a 1-point gain in eighth grade reading.

In a letter obtained by Chalkbeat, NCES associate commissioner Peggy Carr responded to White. She wrote that students nationally did fare worse on digital tests compared to paper, with fourth-graders seeing scores dip more than eighth graders. Those differences — the “digital penalty” — were accounted for with a national adjustment when NCES reported the latest results, Carr notes.

After performing those adjustments, Carr wrote, “For Louisiana … we found no statistically significant differences between the [digital] and [paper] results.” In essence, NCES looked at whether that digital penalty was worse in Louisiana — based on its students who took the paper tests — than in the country as a whole, and finds little evidence of that.

White points out that there were some modest differences in his state, even if they weren’t statistically significant. If the results were based on the the paper exam, the state’s fourth-grade reading scores would only have declined by 1.6 points, rather than 4.6; in eighth grade math, a nearly 2-point decline would have been a half-point gain.

“There is certainly some caution with which we need to view results,” White told Chalkbeat. “My point is you’ve got variation.”

Yet in fourth grade math and eighth grade reading, Louisiana’s digital penalty was actually modestly smaller than the country as a whole, going against White’s concern that his state was particularly affected

The extent to which different states had varying digital penalties was also minimal, inconsistent, and usually not statistically significant, Carr told reporters ahead of the scores’ release.

But an analysis conducted by Johns Hopkins University’s Institute for Education Policy found that states where students had prior experience taking state exams online saw slightly larger gains — at most, 2 points — on three of the four tests.

The results only show that having taken online tests is correlated with those gains; it doesn’t prove that explanation.

Carr declined to comment on the Johns Hopkins report since it had been conducted before the NAEP scores were formally released. White said that he had shared results with the Hopkins researchers, with the stipulation that they honor NCES’s release timeline.

Read Carr’s response in full:



Silver Lining Playbook

Memphis’ youngest students show reading gains on 2018 state tests — and that’s a big deal

PHOTO: Caroline Bauman
A student works on reading comprehension skills at Lucie E Campbell Elementary School in Memphis and Shelby County Schools.

Those working to improve early literacy rates in Shelby County Schools got a small morale boost Thursday as newly released scores show the district’s elementary school students improved their reading on 2018 state tests.

The percentage of Memphis elementary-age students considered proficient in reading rose by 3 points to almost one-fourth of the district’s children in grades 3 through 5. That’s still well below the state average, and Superintendent Dorsey Hopson said “we obviously have a long way to go.”

PHOTO: Caroline Bauman
Superintendent Dorsey Hopson has overseen Tennessee’s largest public school district since 2013.

Strengthening early literacy has been a priority for the Memphis district, which views better reading skills as crucial to predicting high school graduation and career success. To that end, Shelby County Schools has expanded access to pre-K programs, adjusted reading curriculum, and made investments in literacy training for teachers.

Hopson said the payoff on this year’s TNReady scores was a jump of almost 5 percentage points in third-grade reading proficiency.

“It was about five years ago when we really, really, really started pushing pre-K, and those pre-K kids are now in the third grade. I think that’s something that’s really positive,” Hopson said of the gains, adding that third-grade reading levels are an important indicator of future school performance.

TNReady scores for Shelby County Schools, which has a high concentration of low-performing schools and students living in poverty, were a mixed bag, as they were statewide.

Math scores went up in elementary, middle, and high schools in Tennessee’s largest district. But science scores went down across the board, and the percentage of high school students who scored proficient in reading dropped by 4 percentage points.

PHOTO: Shelby County Schools

Hopson said he was unsure how much the reading scores of older students — all of whom tested online — were affected by technical problems that hampered Tennessee’s return this year to computerized testing.

“From what people tell me, kids either didn’t try as hard in some instances or didn’t take it seriously,” Hopson told reporters. “We’ll never know what the real impact is, but we have to accept the data that came from these tests.”

But students in two of the district’s school improvement initiatives — the Innovation Zone and the Empowerment Zone — showed progress. “We’re going to double down on these strategies,” Hopson said of the extra investments and classroom supports.

In the state-run Achievement School District, or ASD, which oversees 30 low-performing schools in Memphis, grades 3 through 8 saw an uptick in scores in both reading and math. But high schoolers scored more than 3 percentage points lower in reading and also took a step back in science.

The ASD takes over schools in the state’s bottom 5 percent and assigns them to charter operators to improve. But in the five years that the ASD has been in Memphis, its scores have been mostly stagnant.

Tennessee Education Commissioner Candice McQueen said she and new ASD Superintendent Sharon Griffin are reviewing the new data to determine next steps.

“We are seeing some encouraging momentum shifts,” McQueen said.

Scores in

After a wild testing year, Tennessee student scores mostly dip — but there are a few bright spots

PHOTO: Getty Images/Sathyanarayan

Student test scores were mostly flat or dipped this year in Tennessee, especially in middle school where performance declined in every subject, according to statewide data released on Thursday.

But there were a few bright spots, including improvement in elementary school English and high school math — both areas of emphasis as the state tries to lift its proficiency rates in literacy and math.

Also, performance gaps tightened in numerous subjects between students in historically underserved populations and their peers. And scores in the state’s lowest-performing “priority” schools, including the state-run Achievement School District, generally improved more than those in non-priority schools.

But in science, students across the board saw declines. This was not expected because Tennessee has not yet transitioned to new, more difficult standards and a new aligned test for that subject. Education Commissioner Candice McQueen said the drops reinforce the need to support science teachers in the shift to higher expectations beginning this fall.

The mixed results come in the third year of the state’s TNReady test, which measures learning based on academic standards that have undergone massive changes in the last five years. The 2017-18 school year was the first under new math and English standards that are based on the previous Common Core benchmarks but were revised to be Tennessee-specific. And in addition to new science standards that kick off this fall, new expectations for social studies will reach classrooms in the 2019-20 school year.

In an afternoon press call, McQueen said “stability matters” when you’re trying to move the needle on student achievement.

“It takes time to really align to the full depth and breadth of these expectations,” she said.

The three charts below illustrate, by subject, the percentage of students statewide who performed on track or better, both this year and last year, in elementary, middle, and high schools. The blue bars reflect the most recent scores.

McQueen acknowledged the good and bad from this year’s results.

“While we’ve focused extensively on early grade reading and are starting to see a shift in the right direction, we know middle school remains a statewide challenge across the board,” she said in a statement.

Tennessee’s data dump comes after a tumultuous spring of testing that was marred by technical problems in the return to statewide computerized exams. About half of the 650,000 students who took TNReady tested online, while the rest stuck with paper and pencil. Online testing snafus were so extensive that the Legislature — concerned about the scores’ reliability — rolled back their importance in students’ final grades, teachers’ evaluations, and the state’s accountability system for schools.

However, the results of a new independent analysis show that the online disruptions had minimal impact on scores. The analysis, conducted by a Virginia-based technical group called the Human Resources Research Organization, will be released in the coming weeks.

Even so, one variable that can’t be measured is the effect of the technical problems on student motivation, especially after the Legislature ordered — in the midst of testing — that the scores didn’t have to be included in final grades.

“The motivation of our students is an unknown we just can’t quantify. We can’t get in their minds on motivation,” McQueen told Chalkbeat on the eve of the scores’ release.

Thursday’s rollout marked the biggest single-day release of state scores since high school students took their first TNReady tests in 2016. (Grades 3-8 took their first in 2017.) The data dump included state- and district-level scores for math, English, science, and U.S. history for grades 3-12.

More scores will come later. School-by-school data will be released in the coming weeks. In addition, Tennessee will unveil the results of its new social studies test for grades 3-8 this fall after setting the thresholds for what constitutes passing scores at each grade level.

You can find the state-level results here and the district-level results here.

Chalkbeat illustrator Sam Park contributed to this story.