Another pause

It’s official! Results from Tennessee’s ugly testing year won’t count for much of anything

PHOTO: Marta W. Aldrich
Gov. Bill Haslam and his education chief, Candice McQueen, speak with reporters Monday about how Tennessee will handle standardized test results this year because of technical problems administering the exams by computer.

Tennessee teachers and school districts can decide how to use this year’s standardized test results and won’t be penalized for low growth scores after another year of problems with the state’s computerized exam.

The state also will shift some responsibilities to a different testing company while deciding whether to extend Minnesota-based Questar’s contract to give the test past November, Education Commissioner Candice McQueen announced on Monday.

In addition, the state has hired two outside groups to sort through all that went wrong with TNReady during almost a month of testing that ended last week. One will look at the validity of the results to see if frequent online interruptions made the scores unreliable. The other will scrutinize Questar’s technology systems to determine why so much went awry.

The actions were announced as McQueen and Gov. Bill Haslam faced reporters together for the first time in the wake of this year’s sloppy return to statewide computerized testing.

After weeks of being on the defensive, Haslam’s administration sought to take control of the situation and emphasize that testing — done correctly — is critical to improving student achievement across the state.

I still have full confidence that testing is the right thing to do,” said the Republican governor, finishing the last year of an eight-year term. I’m frustrated like everybody else that we had issues with the online portion of this. But having said that, do I think the test is a good test? I do.”

Haslam also said Tennessee must forge ahead with computerized testing.

We’re one of only 10 states that has not already moved [completely] to online testing. And so it’s not just that’s where the world is going; that’s where the world is. And our students have to be prepared,” he said.

The decision to shield students, teachers, and schools from accountability for poor growth scores falls in line with emergency legislation passed by state lawmakers last month as reports of TNReady’s technical problems escalated. After weeks of studying the two new laws, McQueen and her team offered their first analysis of what the legislation means:

Teacher evaluations. The state still plans to include student growth scores in evaluations, but each teacher will have “complete control to nullify” that portion if they choose to rely solely on other measures, McQueen said.

Student grades. Local school boards will decide whether to incorporate TNReady scores into this year’s final grades. Many districts already have begun that process, and most are opting to exclude the results this year.

School ratings. Tennessee’s A-F rating system will not launch this fall as scheduled, although the state still will publish the achievement results that would have gone into them.

Priority schools. As planned, the state will release its “priority list” this fall of the 5 percent of lowest-performing schools, but this year’s test results will not be a factor. Instead, the list will be based on two years of previous scores for high schools and one year for lower-grade schools. “We will not be moving any schools based on that data into the Achievement School District,” McQueen said of the state-run turnaround program that takes over local schools and assigns them to charter operators.

Whether the adjustments put Tennesseee out of compliance with federal law remains to be seen, though. The 2015 Every Student Succeeds Act requires that student achievement — as measured by tests like TNReady — be part of each state’s plan for holding struggling schools accountable.

“We’ve been in lots of conversations with the U.S. Department of Education, and they are continuing to work with us on that,” McQueen acknowledged. “So what we will do is create something called a comprehensive support list, which is required under the Every Student Succeeds Act. That will include this year’s data, but there will not be any adverse action taken based on that comprehensive school list.

"It's important for all of us that we get this right. "Gov. Bill Haslam

After three years of testing problems, McQueen announced that Tennessee will create a “TNReady Ambassadors” program to improve customer service and will hire a full-time overseer to work with testing coordinators at the district level.

We did not have our expectations met in terms of customer service from Questar,” she said.

To that end, the state is reviewing its annual $30 million, two-year contract with Questar that expires on Nov. 30.

Haslam said changing companies in the middle of a school year wouldn’t be seamless because Questar will begin testing high school students on non-traditional block schedules this fall. There’s a little bit of a practical problem switching vendors right in the middle of that, so it’s part of negotiations we’re in the middle of,” he said.

McQueen praised Educational Testing Service, the New Jersey-based company that will take over the test’s design work while Questar focuses on test delivery. Also known as ETS, the vendor has had contracts with Tennessee since 2015 to create the state’s social studies and science tests, and to design many of its teacher certification exams. (ETS also owns Questar. Read the details here.)

It is a vendor that is well-known. It has a reputation for very high-quality work in terms of how they design tests,” she said.

Deeper dive: By getting testing wrong again, will Tennessee undo what it may be getting right?

What went down

‘There was no cyber attack,’ investigator says of Tennessee’s online testing shutdown

PHOTO: Manuel Breva Colmeiro/Getty Images

Questar’s unauthorized change of an online testing tool — not a possible cyber attack, as earlier reported by the company — was responsible for shutting down Tennessee’s computerized exams on their second day this spring, the state’s chief investigator reported Wednesday.

An independent probe determined that “there was no cyber attack,” nor was any student data compromised, when thousands of students could not log onto the online exam known as TNReady on April 17.

Instead, investigators said, Questar was mostly responsible for this year’s testing miscues. The main culprit was a combination of “bugs in the software” and the slowness of a computerized tool designed to let students turn text into speech if they need audible instructions.

Comptroller Justin P. Wilson reviewed early findings of his office’s internal review and the external investigation by a company hired by the Education Department during a legislative hearing in Nashville.

Education Commissioner Candice McQueen also told lawmakers that Tennessee is docking Questar about $2.5 million this year out of its $30 million contract because of the online problems that plagued many students and schools during the three-week testing window.

Payments being withheld are punitive, as well as to cover the state’s costs to address the problems, she said, adding that other discounts could follow.

Last week, McQueen announced that the state plans to launch a new search this fall for one or more testing companies to take over TNReady beginning in the 2019-20 school year. She said a track record of successful online testing is a must.

The text-to-speech tool worked fine last fall when a smaller number of high school students tested online. But the state said Questar made a “significant and unauthorized change” to that feature before the launch of spring testing that affects the vast majority of Tennessee students.  

“We now know this decision led to the severity of other issues we experienced during online testing,” the Education Department said in a statement.

House Speaker Beth Harwell and Rep. Jeremy Faison asked the comptroller to review the state’s contract with Questar, particularly related to reports of a possible cyber attack. Wilson’s office also looked into other technical snafus that disrupted student testing for days, prompting the legislature to pass emergency laws that make this year’s scores inconsequential.

“We believe that the student testing issues occurred primarily because of how Questar set the student assessment system up to work,” said Brent Rumbley, the comptroller’s information systems audit manager.

Education Commissioner Candice McQueen testifies during Wednesday’s hearing, where specialists in the state comptroller’s office also testified.

On the second day of exams, Rumbley said, those issues manifested themselves in a suspiciously high volume of internet traffic to the testing platform.

“That’s what led the Department of Education and Questar to believe that there may have been a cyber attack,” he told lawmakers. “This traffic eventually shut the system down.”

Even though Questar upgraded the processing capability of its equipment in response, students and educators continued to report problems logging in, staying online, and submitting tests until Questar turned off the text-to-speech tool beginning May 1.

The comptroller’s office also found that Questar was ill-prepared to handle the fallout from the technical glitches. For instance, the company struggled to manually recover the high number of tests that students couldn’t submit online. And school personnel calling the customer service line experienced wait times as long as 60 minutes, prompting many to just hang up.

New details emerged Wednesday about other testing problems, too.

On April 25, a Questar employee “inadvertently overrode” custom rosters statewide that allowed schools to match students with available testing devices. “As a result, teachers and test coordinators had to scramble to get students the tests they should take,” Rumbley said.

The next day, more problems erupted when an internet cable was severed by a dump truck in a traffic accident in Hawkins County.

“According to the vendor that manages the fiber optic line, 21 districts were without internet from approximately two to four hours,” said Rumbley, adding that neither Questar nor the department could have prevented the outage that day.

Lawmakers will get an expanded look at the Education Department and its testing program in November when Wilson’s office presents the results of a year-long performance audit, along with findings from a massive survey of Tennessee educators about TNReady.

The two-hour hearing gave lawmakers a platform to take jabs at McQueen and her department for their handling of testing.

Rep. Bo Mitchell admonished the Education Department for tweeting on the second day of testing that Questar “may have experienced a deliberate attack” that morning.

“This gets into the public trust and throwing out information to the public from the Department of Education that the failure was a hack … Whose decision was that to put that out into the public domain without any proof?” asked Mitchell, a Democrat from Nashville.

McQueen clarified that the department never used the word “hack,” but reported that the testing system was experiencing a “pattern of data that was consistent with a cyber attack.” The description was based on what was known as the time, she said.

Sen. Janice Bowling, a Republican from Tullahoma, said Questar’s $2.5 million penalty “seems like a smack on the wrist” given the disruption caused by the company’s mistakes.

McQueen responded that the state is withholding almost $11 million invoiced by Questar for online testing as it continues negotiations. She added that the state’s biggest testing expenses stem from printing and transit costs for paper materials used by about half of its students this year. The state is transitioning to computerized testing and has decided to slow the switch for a second time in the wake of this year’s challenges.

Justin P. Wilson

Questar officials told Chalkbeat last week that the company plans to pursue the state’s new contract next year, but Rep. Craig Fitzhugh told McQueen that he doesn’t want the Minnesota-based company involved after it completes its current contract.

“I don’t think we can let Questar get in the ballgame again,” said the Ripley Democrat.

The proposal will be competitively bid, said Wilson, adding that Questar’s past performance will be taken into account.

For more on how Tennessee got here, read why state lawmakers share blame, too, for TNReady testing headaches.

Splitting duties

Tennessee lawmakers OK shifting $12.5 million in TNReady testing work from Questar to ETS

PHOTO: Marta W. Aldrich
Tennessee State Capitol

Tennessee is banking that two companies are better than one when it comes to fixing the state’s troubled standardized testing program.

The legislature’s fiscal review committee gave its blessing Wednesday to hiring New Jersey-based ETS for some chores previously handled by Minnesota-based Questar.

Education Commissioner Candice McQueen said the shift will allow Questar to focus on delivering and scoring the state’s TNReady tests in math and English language arts — tasks that the company has struggled with the last two years.

It also will consolidate all of the state’s test design work with one company since ETS, also known as Educational Testing Service, already develops the questions, instructions, and materials for Tennessee’s social studies and science exams.

Beginning July 1, the change will add up to $12.5 million to ETS’ existing $25 million contract with the state Department of Education that runs through September of 2020.

State officials expect the extra money will be offset by re-negotiating down the cost for Tennessee’s $30 million annual contract with Questar, whose oversight of online testing this spring was plagued by interruptions from a string of technical problems.

McQueen told lawmakers that both companies have done solid work creating test questions and aligning them with Tennessee’s new academic standards — but that ETS has managed that task better.

“They have a much longer history in test development and design,” she said. “They do all our teacher licensure exams. They work across many states. … We’ve had positive interactions with them.”

Splitting the TNReady work among two companies marks a departure for the state Department of Education.

Tennessee has used one testing company at a time since launching the new test in 2016. The first was North Carolina-based Measurement Inc., which McQueen fired over online failures that led to the cancellation of most exams that first year. The state then hired Questar to take over and, except for some scoring problems, TNReady went better in 2017. But this school year, Questar had significant challenges with computerized exams as the state returned to online testing statewide for its older students.

PHOTO: TN.gov
Education Commissioner Candice McQueen has been under fire for her oversight of the state’s standardized test, which has had a string of problems since its 2016 rollout.

“Testing, we’re learning, is very complex in terms of the number of things that one company is expected to do,” McQueen said.

Beyond creating the questions, testing companies provide Tennessee’s online and paper tests, score the answers, analyze the results to make sure they’re reliable, and report the data.

“We know that certain companies do some pieces much better than others,” McQueen said.

Questar’s contract ends in November but, even with this year’s problems, likely will be extended through next spring since the company will oversee testing this fall for high school students on nontraditional block schedules. Earlier this month, Gov. Bill Haslam said switching companies in the middle of a school year would present a “practical problem.”

McQueen and her staff faced stern questions about the ETS contract before gaining approval of the fiscal review committee. Among members’ concerns: the cost of testing services, whether the change will smooth out testing next year, and ETS’ purchase of Questar last year. (For details on that deal, read TNReady’s new testing company also owns the old one.)

“They are separate companies,” McQueen said of Questar becoming a subsidiary of ETS. “They have separate contracting, separate contract management, separate CEOS. We were working with ETS before we got in a relationship with Questar.”