Last-minute questions about differences in the 2015 online and paper-pencil ISTEP tests postponed an Indiana State Board of Education vote to set passing scores on the exams today.
The delay put off a discussion about big drops expected in the number of students who passed ISTEP last year if the board adopts the recommended passing scores. The passing rate declines are connected to tougher new academic standards, which made for more difficult ISTEP tests.
Board members, who often have been at odds with state Superintendent Glenda Ritz and the Indiana Department of Education staff, quickly complained that the latest delay was another example of the Ritz’s team failing to provide timely information to the board.
This delay is the second for ISTEP scores after the testing company that creates the exams, California-based CTB, reported problems with short-answer questions back in August.
“In my view this was avoidable,” board member Gordon Hendry said. “We could’ve voted on cut scores today if the information was delivered like it should have been.”
But Danielle Shockey, deputy state superintendent, said nothing was withheld from the board
A draft study doesn’t actually need to be done first, she said. The process used to come up with the recommended passing scores could still be approved, and technical adjustments could come later.
“It is the same valid work we’ve done previously for other tests,” Shockey said. “There is absolutely no reason to pause or delay the test cut-score setting today.”
ISTEP scores factor into many critical statewide decisions, such as school A-to-F grades and teacher evaluations, as well as teacher pay decisions. At last month’s board meeting, the Indiana Department of Education said school grades wouldn’t be released until January 2016.
The comparison study is very technical. It details statistical tests performed on test scores and questions from both the online and paper-pencil tests. The findings seem to suggest that some questions on certain tests are harder on the online test than the paper-pencil test and vice versa, particularly in the math section.
“The concerns center on a report by the Department of Education from October 2, but was not provided to the test’s Technical Advisory Committee, a team of state and national experts hired to monitor the testing process, and the State Board of Education until Tuesday night,” said a statement from the state board. “That report raises serious questions about potential differences in difficulty between the online and paper/pencil tests.”
Comparison studies are not unusual, in Indiana or in general. They are done to see whether different test formats affect the validity of the test scores.
But the state board wants its own test experts to review the study, and it says the department didn’t fulfill its end of the bargain. Board member Sarah O’Brien said she requested this study back in July and was told by the education department that it would be finished before work on passing scores started.
“In order for us to justify such a huge decision for the field, we need to be able to say we have this unbiased information saying we are on-track,” said O’Brien, a teacher from Avon. “I don’t feel we have that.”
Karla Egan, a test expert on the technical advisory committee that helps oversee the score-setting process, said the comparison study doesn’t necessarily show something is wrong. But the test experts should have it before they approve recommended cut scores.
“We don’t know that there’s a problem,” Egan said. “We just need more time.”
State board test director Cynthia Roach said comparison studies should, and typically have been, completed before cut-off scores are created to begin with. When asked why the process continued without the study, state board spokesman Marc Lotter said they were assured the tests were comparable by a CTB representative, but no actual data was given.
Michelle Walker, the department’s test director, said based on the draft study by CTB, she’s confident the paper-pencil and online tests are comparable and will prove valid. The state board’s review of the process doesn’t have to hold things up.
“The draft came to us on (Oct. 2) so we could discuss it with CTB last week, which we did do,” Walker said. “If they want feedback … I think that’s fine. I think that’s a third-party, secondary piece in the process.”
The board was expected to vote today on ISTEP passing scores, which projections said could drop by about 16 percentage points in English and 24 percentage points in math compared to 2014. The drops are worse than in prior years, as ISTEP scores have rarely swung up or down by more than a few percentage points since 2010.
Part of that difference is because the 2015 ISTEP test was written to measure new, more challenging academic standards. For that reason, lower scores were expected, and some educators and advocates say other states that have switched standards saw similar changes.
Ritz said it’s possible the board could vote on the recommended passing scores in two weeks when it holds a special meeting to discuss new proposed high school diplomas on Oct. 28.