The sole exam that determines admission to New York City’s elite specialized high schools generally predicts which students are likely to be successful in high school, according to a study the city has kept secret for years.
The study, which was completed in 2013 and obtained by Chalkbeat through a public records request, found that the high-stakes Specialized High School Admissions Test, or SHSAT, has a “strong positive predictive relationship” with student achievement during the first two years of high school, measured by grades, scores on AP tests, and Regents exams.
The test has been the subject of a heated debate ever since Mayor Bill de Blasio proposed eliminating it earlier this summer to make the schools more racially representative of the city’s student population.
The test itself has been subject to remarkably little formal evaluation, including whether it accurately selects students who are likely to flourish at specialized high schools. The 2013 report is the first of its kind to directly look at that issue.
Still, the study doesn’t address key questions about whether the SHSAT is any better at predicting student success than the alternative system de Blasio put forward. And it can’t get at the heart of the debate about the importance of diversifying the elite schools.
The study uses data from every single eighth grade student who took the SHSAT between 2005 and 2009, looking at whether a student’s score seemed to predict early success in high school. It finds a relatively strong relationship between SHSAT scores and early high school performance.
Critics of de Blasio’s plan to get rid of the test will likely point to the research as evidence the exam is worthwhile and shouldn’t be abandoned. After all, it selects students who are likely to be successful at schools such as Stuyvesant, Bronx Science, and Brooklyn Tech.
But two outside education researchers said the study doesn’t actually offer any evidence about whether the SHSAT is the best admissions method, since it does not look at whether alternative methods would also yield similarly strong students.
“It tells us something we already knew: Kids who do well on the SHSAT do well in high school,” said Aaron Pallas, a researcher at Columbia who reviewed the study at Chalkbeat’s request. “But it doesn’t tell us what is the best combination of factors that predict who might do well in an exam school.”
The education department itself also downplayed the study’s findings.
“What the validity study misses is the kid who didn’t do as well on the test, or didn’t take it, but still stands an excellent chance of being successful in these high schools if they had the opportunity,” education department spokeswoman Toya Holness wrote in a statement.
The education department has previously declined to release the report, which was commissioned in response to a civil rights complaint lodged by the NAACP and completed in 2013, before de Blasio took office the following year.
The report does not look at longer-term outcomes, such as graduation rates or college enrollment, but previous research has found that the SHSAT does not predict college attendance or graduation. The study also can’t account for changes to the SHSAT, which was redesigned last year to more closely align with material that students learn in middle school. And it doesn’t offer demographic breakdowns by race or socioeconomic status.
De Blasio has proposed scrapping the test entirely in an effort to desegregate the specialized high schools, saying the process is unfair to students whose families can’t afford test prep. Though 67 percent of the city’s public school students are black or Hispanic, just 10 percent of offers to attend specialized high schools go to black or Hispanic students.
Instead, de Blasio has proposed a University of Texas-style system that would admit the top 7 percent of students at every middle school, which would significantly increase the number of black and Hispanic students who are admitted.
Critics, including vocal alumni groups, parents, and some politicians, have argued that the mayor’s plan would water down the schools’ academic rigor, but an analysis conducted by the city has found the proposal would not have a sizeable impact on incoming students’ middle school test scores or GPAs.
Overall, Pallas says the new findings don’t have strong policy implications for the broader debate.
“This is a primitive analysis,” he said. “This is going to be used as a Rorschach test where people will see what they want to see in it.”