One day after the U.S. Department of Education told states that they must give standardized tests this year, a number of state officials gathered virtually to hear from leading testing experts.
Those experts offered a warning: be careful how you interpret this year’s results.
Derek Briggs, a University of Colorado professor, presented a hypothetical. Imagine some schools see their test scores hold steady, while others see declines. The obvious conclusion would be that certain schools did better than others helping their students through the pandemic.
That, Briggs explained during a panel organized by the Council of Chief State School Officers, could be entirely wrong. Instead, the results might say more about which students from each school actually took the tests.
“In all of these situations, it depends on who is being included and who’s being excluded,” he said.
Usually, participation isn’t such a big problem. What distinguishes state assessments from exams given by individual teachers is the fact that nearly all students take the same tests around the same time under similar conditions.
This year, though, there’s a good chance a big chunk of students will opt out of the exams, among other potential changes. Experts say that means the tests likely won’t be able to offer the kind of complete or comparable data policymakers rely on in a typical year.
Scott Marion, executive director of the Center for Assessment, an organization that works with the technical advisory groups that oversee tests for a number of states, said experts at these meetings have been concerned.
“Every one of them is nervous about the validity of the results,” he said. “You can pretend it’s comparable, but it’s not going to be comparable.”
Why it matters how many students show up to take the tests
A critical challenge will be getting students to actually take the exams. An estimated one in three U.S. students is attending a school operating entirely remotely, and many families where schools are offering in-person learning have chosen not to send their children back into buildings.
For states that choose a straightforward approach — testing students in person this spring — participation is likely to be lower than normal as families opt out over safety concerns.
“State testing isn’t one of those things that if you build it, they will come,” said Marianne Perie, a testing consultant who works with several states. The existence of the exam “doesn’t mean parents will send their kids to school.”
A lot may depend on messaging from state officials and local educators, some of whom were hoping tests would be cancelled altogether. Georgia’s state superintendent, for instance, is telling district leaders not to require students learning virtually to show up for tests.
“I’m not going to be out there trying to convince people to not show up,” said Justin Parmenter, a middle school language arts teacher in Charlotte, North Carolina. “On the other hand, I just don’t feel it’s appropriate for me to try to convince people to come in during a year when we’ve made clear from the beginning that we’re going to be flexible.”
Federal officials this week implicitly acknowledged that participation could drop, telling state leaders the department planned to waive a provision saying that 95% of each school’s students must take the test or face potential state penalties.
If many students don’t take the exams, that obviously means there won’t be data on their academic performance. But the issue goes beyond that: If some groups of students are more likely to miss the exam than others, that could skew the overall picture.
Evidence of those disparities has already cropped up, likely because low-income students and students of color are more likely to be learning remotely. When Ohio gave third graders reading exams in person this fall, participation rates fell by 9 percentage points among white students but 29 points among Black students, for example.
“The very student groups that have faced the greatest historical disadvantages will be systematically underrepresented in any near-term statewide assessment administration,” Pennsylvania Education Secretary Noe Ortega said in a draft letter to federal officials.
That doesn’t mean the data is useless, but it’s not clear how reliably it can be used to determine which schools or districts need extra help and which have been most successful during the pandemic.
“The individual test results for the individual students will be valid, but trying to make any kind of meaningful conclusions to drive policy is what’s going to be suspect,” said Perie.
There are alternatives, but each creates new problems
Federal officials have signaled to state leaders that they could change how testing works this year to boost participation rates.
One option is for states to allow students to take exams remotely. Perie and Marion said they were aware of only a handful of states considering this option, which raises a host of other issues, including making sure that students have access to reliable internet, preventing students from getting help from family members, and ensuring students with disabilities get the accommodations they’re entitled to by law.
“All these things that we look at in standardized testing would not be standardized,” said Perie.
Another option: Giving the tests next fall, when more students are likely to be back in school buildings for regular instruction. Pennsylvania is pursuing this option as a choice for districts.
But if some districts take the exam in the spring and others take it in fall, the results will be difficult to compare. Delaying tests also means a delay in results, limiting their usefulness to policymakers and educators.
“That to me raises the theory-of-action question in more stark terms,” said Marion. “Is it just an intellectual exercise at that point?”
Meanwhile, officials in a number of states, including Colorado and Michigan, are considering a third option: giving students an exam that each district chooses, like the NWEA MAP. That would allow districts to use tests they were giving anyway, simplifying the process.
But that approach also limits the ability to compare results, which is one of the key arguments for standardized testing. “By design, these local assessments do not hold all students to the same standards and expectations,” wrote a coalition of civil rights, business, school reform, and other groups opposed to this idea.
A final way to account for the missing students involves making statistical projections. A state could use data from students who do take the tests to make an educated guess about the results for seemingly similar students who don’t take the tests. But that is an inexact science.
“My concern is the kids who have never shown up, the kids we’ve lost in 2021,” said Perie. “That, we won’t have any data from, and there’s nothing we could do to model how those kids would have done.”
