A machine-scored writing exercise introduced this year as part of the admissions process for five of Philadelphia’s top public high schools is being misused, an education professor says.
Joshua Wilson, an associate professor at the University of Delaware who studies automated essay scoring, said the writing tool is meant to identify struggling learners and inform classroom instruction, not make high-stakes decisions about students’ futures.
“No one has done research on whether it can be used to make placement decisions,” Wilson said about the writing tool, MI Write, which is a product made by Measurement Inc., based in Durham, N.C. Using it this way is a “mistake,” he said.
“I believe the use of Mi Write in this context is very problematic and should be reconsidered,” Wilson told the Philadelphia school board at its Thursday meeting.
In response to Wilson’s criticism, district spokesperson Monica Lewis issued a statement to Chalkbeat saying that the district had been “assured by the vendor that using this tool to score essays as part of the application process would be appropriate.”
Lewis said that one of the five schools, Parkway Center City Middle College, had for several years been using a computer-graded writing sample for admissions with “positive feedback.”
Measurement, Inc. did not immediately respond to an email request for comment. But a document on its website says MI Write scores should not be used to grade students.
“It would be very unfair to assign a grade to a student based on its evaluation of an essay,” the document says. “One student might benefit from transferring [the scoring engine’s] scores to grades, while another might be penalized. MI Write was designed to help students practice their skills in writing and to improve them based on feedback.”
This year, the school district overhauled its selective admissions process in an effort to improve access for traditionally underserved students. Instead of giving individual school leaders final say over who is admitted, all students who meet the qualifications are entered into a lottery.
Each school can still set standards for grades, attendance, and behavior. But missing the cutoff on the machine-scored writing sample can eliminate a student from the lottery.
Concern about the writing test is only one of several issues being raised about the new process. Several parents and students registered to speak at the meeting on this topic.
Black and Latino enrollment at the district’s two most coveted and rigorously selective schools, Central High and Masterman Laboratory and Demonstration School, has declined in recent years, an issue highlighted during the national protests that followed a police officer killing George Floyd last year. Alumni, students, and some faculty at the two schools demanded that the district act. Black enrollment at both schools is below 20%, in a district where more than half of the students are Black.
The goal of the new system, officials said, is to address racial disparities in admissions and open opportunities to more students by attempting to minimize any bias that could occur as school leaders shaped their next class.
“We worked hard to avoid data points that involved human assessment,” said Karyn Lynch, the district’s chief of student support services, when she announced the new process in October, on the day that the application window for fall 2022 admissions opened.
At Thursday’s board meeting, she offered a strong defense of the new process “as eliminating privilege so other students can finally have the opportunity they had not had before. We have also heard from many excited students who are suddenly are aware of a school selection process where they didn’t participate in the past but are participating now.”
As part of the new system, the district also gave preference to students living in five ZIP codes where students are underrepresented at top schools.
Wilson, the professor, called the effort to make the system more equitable “a noble goal,” but said the question was “how to get the action to match the intention with a more valid way of doing it.”
Besides Central and Masterman, the other three schools requiring the writing test are Academy at Palumbo, Parkway Center City, and Carver High School of Engineering and Science. Carver and Parkway Center City are majority Black, while Palumbo is one-third Black and one-third Asian American.
Before the change, students were admitted to the five schools based on grades, attendance, behavior, and a writing sample.
Standardized test scores were a big factor before the pandemic, but they haven’t been administered since spring 2019. Officials eliminated their use for this cycle, but have not said whether they will resume using them in the future. Standardized test results typically correlate with socioeconomic status, putting underserved Black and Latino students at a disadvantage.
The MI Write test used this year gives students a prompt, mostly on topics that don’t require detailed background knowledge. Questions might include “What is your favorite subject and why?” or “If you could go back in history and meet someone, who would that be?” Students have 90 minutes to write an essay that is immediately scored on a scale of six to 30.
To qualify for Palumbo, Carver and Parkway Center City, the student needs a score of 17. For Central and Masterman, the score is 22, according to the district.
Wilson raised concerns about how the district determined the cutoff scores on the MI Write test for admission. “Where’s the research that validates the cut scores?” he said. “I think Philadelphia has the burden of showing the proof that these cut scores are appropriate.”
Wilson said that 17 “is just below the median, and probably not that high a bar.” In a 90-minute sample, he said, “it would probably be achievable for most kids unless they really lack basic skills.”
At the board meeting, member Mallory Fix-Lopez said she had heard that some students are getting fractional scores, such as 21.9, and asked Lynch if the scores would be rounded up so students would make the cutoff for Masterman and Central. Lynch said no.
Fix-Lopez said the students she spoke with who missed the cutoff by tenths of a point are Black and Latino, “the population we were particularly looking at to address equity issues.” She said she would like to see demographic data on otherwise qualified students who fell into that kind of gap.
Student Nora Ouarirdi told the board she scored a 21.1. “I cried,” she said. “For students worked incredibly hard, this doesn’t seem particularly equitable.”
Wilson also noted that the instrument doesn’t score for content. “I can write a beautifully written response off-topic and get a good score,” he said.
Students take the computerized exercise in class, and immediately learn their score. Students are taking the test now; the testing window is between Nov. 29 and Dec. 18.
Timothy Boyle, principal of Science Leadership Academy Middle School, said that 43 of his students took the test last week, and almost all scored above 17. Based on what he knew about the students, he said he wondered what the scores actually revealed about them.
One straight-A student scored a 21, one point shy of the score that would make them eligible for Central or Masterman. Another student, with a 1.5 grade point average, scored a 20, putting them above the cutoff for three of the five schools, though they might be ineligible based on other factors. “How does this result in more fairness and more opportunity?” Boyle asked.
The scoring system is based on an algorithm derived from what trained human raters produced based on the six-trait writing model, which includes rubrics for ideas, organization, sentence fluency, word choice, and style conventions. Each trait can receive a score between 1 and 5, adding to a maximum score of 30.
Wilson said the main use and benefit of machine-scored tests is to encourage teachers to assign more writing because it helps them to pinpoint student weaknesses and guide improvements quickly. “Scoring and evaluating writing is so time consuming,” he said.
Boyle also applauds the effort to make the system more equitable. But he said if district leaders suspect bias on the part of its educators as a reason for the disparities in the demographics at its most selective schools, it needs to tackle that issue directly.
“They say they don’t know if this [new process] is going to work,” he said. “It’s like they are throwing stuff at the wall, and confusing action for strategy.”
The high school selection process includes three tiers — the five “criteria-based” ones at the top, “citywide admission” schools with a particular theme or focus, and those with catchment areas that have special programs.
In the statement, made in response to Chalkbeat’s questions, Lewis said that this year, the district received more high school applications compared to last, suggesting that the effort to broaden access is working. She said 15,382 students — from both inside and outside the district — submitted 62,591 applications (each student can choose five schools), which amounts to “365 more students participating in the process than last year.”
Lynch and Superintendent William Hite, who will end his 10-year tenure in August, have said that the intent is to continually refine the process.
“We know that change can be difficult and we also know that the time for equity-focused change is now,” the district’s statement said. “This year’s changes are a starting point” that will lead to “further enhancements...as we move toward becoming a stronger and more equitable school district.”
