Are Children Learning

Glenda Ritz's team fires back at Pence, says no changes planned for ISTEP — yet

PHOTO: Scott Elliott
Indiana State Board of Education members Cari Whicker and Brad Oliver with state Superintendent Glenda Ritz, pictured at December's meeting.

A spokesman for state Superintendent Glenda Ritz today rejected Gov. Mike Pence’s call to shorten the state ISTEP exam, calling his claims that she mishandled the test “personal and false attacks.”

Spokesman Daniel Altman said Pence had gotten bad information from the Indiana State Board of Education and its staff suggesting Ritz had failed to keep the board informed.

“It’s either politically malicious or a serious sign of staff incompetence,” Altman said. “My hope is that he was simply misinformed.”

But a state board member said the evidence suggests Pence was right.

“That’s just not true,” board member Brad Oliver said of Altman’s claims that the board understood months ago the test would be longer.

Pence on Monday blasted Ritz for leading the development of an ISTEP test that could take students twice as long to complete as last year and issued an order that it be shortened.

But Danielle Shockey, Ritz’s deputy superintendent, said the order would likely have no effect and that the test could not be significantly cut down unless the legislature changes requirements in state law or the U.S. Department of Education offered special freedom from its rules that it has so far denied.

“There is nothing in that order that would cause us to pause,” she said.

The state board will hold a special meeting Friday, but Shockey said there is no plan right now to change course on ISTEP.

“We are planning to move forward with ISTEP as planned, given that is the expectation of the state and the federal government,” she said.

The state board, however, might have other ideas. Oliver said ISTEP must be changed now.

“At this point, let’s just get it fixed,” he said. “I’m tired of this carrying on. It is unacceptable and unreasonable to do this.”

Pence today said he had hired a testing expert, Michigan State University professor Edward Roeber, for $22,000 and had contacted CTB/McGraw-Hill, the California company that makes ISTEP, to explore ways to cut the test down.

“I look forward to your cooperation as we work expeditiously to find a solution that serves Hoosier students, families and educators,” Pence wrote in a letter to Ritz today informing her of Roeber’s hiring.

An irritated Pence told reporters in a quickly arranged press conference on Monday he was shocked to learn ISTEP could take some students more than 12 hours to complete, about twice as much time as was allotted for the series of tests last year.

Pence said the state board was kept in the dark about the fact that the test would be so long, and he demanded action to reduce the testing time.

But Shockey and Altman said Ritz only learned of the time needed for testing from CTB/McGraw-Hill on Jan. 23. She notified schools on Jan. 26 and told the state board at its Feb. 4 meeting.

Shockey said although the exact time needed for ISTEP came out last month, the state board knew last August that it would be much longer. The board heard a presentation from CTB/McGraw-Hill on Aug. 6, and handouts showed a big jump in the number of test questions. The board was cautioned that more questions would make the test take longer, Shockey said.

Oliver responded that the minutes of that meeting prove Shockey wrong. He said the board was told the opposite by Michelle Walker, the state education department’s testing chief, in that August meeting: that it was too early to tell how long ISTEP would be.

The minutes from the meeting, available on the state board’s website, state: “Dr. Oliver asked how much longer the test would be this year. Dr. Michelle Walker said that this is something that they are still looking at in conjunction with CTB, and since the items are still being reviewed, there is no way to know just yet.”

Oliver said that proved what the state board has been saying — that there was no indication before last week that the test would could take twice as long.

“I did ask this question in August, they said they didn’t know,” he said. “That’s fine. This was an agenda item every month. They could have told us any time.”

ISTEP is longer this year for several reasons.

For one thing, the state changed course over the past two years on academic standards, dropping out of Common Core and creating its own standards that the state board approved last April.

That meant Indiana needed a new test test to match its more rigorous expectations for what children should know. In a way, this year’s ISTEP is three tests in one. Part of the test is similar to past exams. Another part includes new questions measuring if students learned the tougher material of the new standards. Finally, there are a series of questions on the 2015 test that won’t count but that are being tried out so they can be used in 2016.

The test also is also longer because it is designed to meet requirements in both state and federal law, Shockey said.

For example, the state requires ISTEP to measure both the level of student skill — such as whether the student passes or fails — and how much they improved during the school year. To measure growth the test must have extra questions that are slightly above and below the student’s grade level to help gauge their gains.

Shockey said Ritz asked federal education officials last year if they could have more time to develop the test, but the answer ultimately was no. Although Indiana made a late change in direction on standards, it would have to test its new standards in 2015 as it had promised, she said Ritz was told.

During those talks, Shockey said, Ritz’s department went ahead and developed a plan to transition in stages to a new ISTEP in 2016, but in April and May both the state board and the federal education department nixed that idea, requiring a quick change of direction.

At this point, Shockey said, the only way to cut the testing time is for the legislature to act. She suggested lawmakers could suspend the third-grade reading test or state fifth- and seventh-grade social studies tests — both of which are state-required but not federally mandated.

The state education department has told schools they can cut down the practice time they had scheduled in advance of ISTEP to save some classroom time. Altman said Ritz would work with Pence, the consultant and the state board to explore other options.

But Shockey said schools should expect to administer ISTEP as planned until they hear differently, and that parents, teachers and students should’t worry.

“We should tell our kids to go in there and do their best, period,” she said.

Surprising report

EXCLUSIVE: Did online snafus skew Tennessee test scores? Analysts say not much

Education Commissioner Candice McQueen will release the results of Tennessee's 2017-18 standardized test this week, but the reliability of those TNReady scores has been in question since this spring's problem-plagued administration of the online exam.

An independent analysis of technical problems that disrupted Tennessee’s online testing program this spring is challenging popular opinion that student scores were significantly tainted as a result.

Education Commissioner Candice McQueen said Wednesday that the disruptions to computerized testing had “small to no impact” on scores, based on a monthlong analysis by the Human Resources Research Organization, or HumRRO. The Virginia-based technical group has expertise in psychometrics, the science behind educational assessments.

“We do believe these are valid, reliable scores,” McQueen told Chalkbeat on the eve of releasing state- and district-level scores for TNReady, the state’s standardized test in its third year.

Here are five things to know as Tennessee prepares to release TNReady scores

The state hired the research group to scrutinize several issues, including whether frequent online testing snafus made this year’s results unreliable. For instance, during at least seven days out of the three-week testing window, students statewide reported problems logging in, staying online, and submitting their tests — issues that eventually prompted the Legislature to roll back the importance of scores in students’ final grades, teacher evaluations, and school accountability systems.

But the analysis did not reveal a dramatic impact.

“For students who experienced the disruption, the analysis did not find any systematic effect on test scores that resulted from lapses in time between signing in and submitting their tests,” McQueen told Chalkbeat.

There was, however, a “small but consistent effect” if a student had to log on multiple times in order to complete the test, she said.

“When I say small, we’re talking about an impact that would be a handful of scale score points out of, say, a possible 200 or 250 points,” McQueen said.

Analysts found some differences in test score averages between 2017 and 2018 but concluded they were not due to the technical disruptions.

“Plausible explanations could be the students just didn’t know the (academic) standards as well and just didn’t do as well on the test,” McQueen said. “Or perhaps they were less motivated after learning that their scores would not count in their final grades after the legislation passed. … The motivation of our students is an unknown we just can’t quantify. We can’t get in their minds on motivation.”

About half of the 600,000 students who took TNReady this year tested with computers, and the other half used paper materials in the state’s transition to online exams. Those testing online included all high school students.

Out of about 502,000 end-of-course tests administered to high schoolers, educators filed about 7,600 irregularity reports – about 1.4 percent – related to problems with test administration, which automatically invalidated those results.

The state asked the analysts specifically to look at the irregularity reports for patterns that could be cause for concern, such as demographic shifts or excessive use of invalidations. They found none.

TNReady headaches started on April 16 – the first day of testing – when students struggled to log on. More problems emerged during the weeks that followed until technicians finally traced the issues to a combination of “bugs in the software” and the slowness of a computerized tool that helps students in need of audible instructions. At one point, officials with testing company Questar blamed a possible cyberattack for shutting down its online platform, but state investigators later dismissed that theory.

While this year’s scores officially are mostly inconsequential, McQueen emphasized Wednesday that the results are still valuable for understanding student performance and growth and analyzing the effectiveness of classroom instruction across Tennessee.

“TNReady scores should be looked at just like any data point in the scheme of multiple data points,” she said. “That’s how we talk about this every year. But it’s an important data point.”

heads up

Tennessee will release TNReady test scores on Thursday. Here are five things to know.

PHOTO: Getty Images/Kali9

When Tennessee unveils its latest standardized test scores on Thursday, the results won’t count for much.

Technical problems marred the return to statewide online testing this spring, prompting the passage of several emergency state laws that rendered this year’s TNReady scores mostly inconsequential. As a result, poor results can’t be used to hold students, educators, or schools accountable — for instance, firing a teacher or taking over a struggling school through the state’s Achievement School District.

But good or bad, the scores still can be useful, say teachers like Josh Rutherford, whose 11th-grade students were among those who experienced frequent online testing interruptions in April.

“There are things we can learn from the data,” said Rutherford, who teaches English at Houston County High School. “I think it would be unprofessional to simply dismiss this year’s scores.”

Heading into this week’s data dump, here are five things to know:

1. This will be the biggest single-day release of state scores since the TNReady era began three years ago.

Anyone with internet access will be able to view state- and district-level scores for math, English, and science for grades 3-12. And more scores will come later. School-by-school data will be released publicly in a few weeks. In addition, Tennessee will unveil the results of its new social studies test this fall after setting the thresholds for what constitutes passing scores at each grade level.

2. Still, this year’s results are anticlimactic.

There are two major reasons. First, many educators and parents question the scores’ reliability due to days of online testing headaches. They also worry that high school students stopped trying after legislators stepped in to say the scores don’t necessarily have to count in final grades. Second, because the scores won’t carry their intended weight, the stakes are lower this year. For instance, teachers have the option of nullifying their evaluation scores. And the state also won’t give each school an A-F grade this fall as originally planned. TNReady scores were supposed to be incorporated into both of those accountability measures.

3. The state is looking into the reliability of the online test scores.

In addition to an internal review by the Education Department, the state commissioned an independent analysis by the Human Resources Research Organization. Researchers for the Virginia-based technical group studied the impact of Tennessee’s online interruptions by looking into testing irregularity reports filed in schools and by scrutinizing variances from year to year and school to school, among other things. (Editor’s note: After this story’s initial publication, Education Commissioner Candice McQueen revealed what the analysis found. Here’s that story.)

4. The reliability of paper-and-pencil test scores are not as much in question.

Only about half of Tennessee’s 600,000 students who took TNReady this year tested on computers. The other half — in grades 3-5 and many students in grades 6-8 — took the exams the old-fashioned way. Though there were some complaints related to paper testing too, state officials say they’re confident about those results. Even so, the Legislature made no distinction between the online and paper administrations of TNReady when they ordered that scores only count if they benefit students, teachers, and schools.

5. Ultimately, districts and school communities will decide how to use this year’s data.

Even within the same district, it wasn’t uncommon for one school to experience online problems and another to enjoy a much smoother testing experience. “Every district was impacted differently,” said Dale Lynch, executive director of the state superintendents organization. “It’s up to the local district to look at the data and make decisions based on those local experiences.”

District leaders have been reviewing the embargoed scores for several weeks, and they’ll share them with teachers in the days and weeks ahead. As for families, parents can ask to see their students’ individual score reports so they can learn from this year’s results, too. Districts distribute those reports in different ways, but they’re fair game beginning Thursday. You can learn more here.