a high-stakes evaluation

The Gates Foundation bet big on teacher evaluation. The report it commissioned explains how those efforts fell short.

PHOTO: Brandon Dill/The Commercial Appeal
Sixth-grade teacher James Johnson leads his students in a gameshow-style lesson on energy at Chickasaw Middle School in 2014 in Shelby County. The district was one of three that received a grant from the Gates Foundation to overhaul teacher evaluation.

Barack Obama’s 2012 State of the Union address reflected the heady moment in education. “We know a good teacher can increase the lifetime income of a classroom by over $250,000,” he said. “A great teacher can offer an escape from poverty to the child who dreams beyond his circumstance.”

Bad teachers were the problem; good teachers were the solution. It was a simplified binary, but the idea and the research it drew on had spurred policy changes across the country, including a spate of laws establishing new evaluation systems designed to reward top teachers and help weed out low performers.

Behind that effort was the Bill and Melinda Gates Foundation, which backed research and advocacy that ultimately shaped these changes.

It also funded the efforts themselves, specifically in several large school districts and charter networks open to changing how teachers were hired, trained, evaluated, and paid. Now, new research commissioned by the Gates Foundation finds scant evidence that those changes accomplished what they were meant to: improve teacher quality or boost student learning.  

The 500-plus page report by the Rand Corporation, released Thursday, details the political and technical challenges of putting complex new systems in place and the steep cost — $575 million — of doing so.

The post-mortem will likely serve as validation to the foundation’s critics, who have long complained about Gates’ heavy influence on education policy and what they call its top-down approach.

The report also comes as the foundation has shifted its priorities away from teacher evaluation and toward other issues, including improving curriculum.

“We have taken these lessons to heart, and they are reflected in the work that we’re doing moving forward,” the Gates Foundation’s Allan Golston said in a statement.

The initiative did not lead to clear gains in student learning.

At the three districts and four California-based charter school networks that took part of the Gates initiative — Pittsburgh; Shelby County (Memphis), Tennessee; Hillsborough County, Florida; and the Alliance-College Ready, Aspire, Green Dot, and Partnerships to Uplift Communities networks — results were spotty. The trends over time didn’t look much better than similar schools in the same state.

Several years into the initiative, there was evidence that it was helping high school reading in Pittsburgh and at the charter networks, but hurting elementary and middle school math in Memphis and among the charters. In most cases there were no clear effects, good or bad. There was also no consistent pattern of results over time.

A complicating factor here is that the comparison schools may also have been changing their teacher evaluations, as the study spanned from 2010 to 2015, when many states passed laws putting in place tougher evaluations and weakening tenure.

There were also lots of other changes going on in the districts and states — like the adoption of Common Core standards, changes in state tests, the expansion of school choice — making it hard to isolate cause and effect. Studies in Chicago, Cincinnati, and Washington D.C. have found that evaluation changes had more positive effects.

Matt Kraft, a professor at Brown who has extensively studied teacher evaluation efforts, said the disappointing results in the latest research couldn’t simply be chalked up to a messy rollout.

These “districts were very well poised to have high-quality implementation,” he said. “That speaks to the actual package of reforms being limited in its potential.”

Principals were generally positive about the changes, but teachers had more complicated views.

From Pittsburgh to Tampa, Florida, the vast majority of principals agreed at least somewhat that “in the long run, students will benefit from the teacher-evaluation system.”

Source: RAND Corporation

Teachers in district schools were far less confident.

When the initiative started, a majority of teachers in all three districts tended to agree with the sentiment. But several years later, support had dipped substantially. This may have reflected dissatisfaction with the previous system — the researchers note that “many veteran [Pittsburgh] teachers we interviewed reported that their principals had never observed them” — and growing disillusionment with the new one.

Majorities of teachers in all locations reported that they had received useful feedback from their classroom observations and changed their habits as a result.

At the same time, teachers in the three districts were highly skeptical that the evaluation system was fair — or that it made sense to attach high-stakes consequences to the results.

The initiative didn’t help ensure that poor students of color had more access to effective teachers.

Part of the impetus for evaluation reform was the idea, backed by some research, that black and Hispanic students from low-income families were more likely to have lower-quality teachers.  

But the initiative didn’t seem to make a difference. In Hillsborough County, inequity expanded. (Surprisingly, before the changes began, the study found that low-income kids of color actually had similar or slightly more effective teachers than other students in Pittsburgh, Hillsborough County, and Shelby County.)

Districts put in place modest bonuses to get top teachers to switch schools, but the evaluation system itself may have been a deterrent.

“Central-office staff in [Hillsborough County] reported that teachers were reluctant to transfer to high-need schools despite the cash incentive and extra support because they believed that obtaining a good VAM score would be difficult at a high-need school,” the report says.

Evaluation was costly — both in terms of time and money.

The total direct cost of all aspects of the program, across several years in the three districts and four charter networks, was $575 million.

That amounts to between 1.5 and 6.5 percent of district or network budgets, or a few hundred dollars per student per year. Over a third of that money came from the Gates Foundation.

The study also quantifies the strain of the new evaluations on school leaders’ and teachers’ time as costing upwards of $200 per student, nearly doubling the the price tag in some districts.

Teachers tended to get high marks on the evaluation system.

Before the new evaluation systems were put in place, the vast majority of teachers got high ratings. That hasn’t changed much, according to this study, which is consistent with national research.

In Pittsburgh, in the initial two years, when evaluations had low stakes, a substantial number of teachers got low marks. That drew objections from the union.

“According to central-office staff, the district adjusted the proposed performance ranges (i.e., lowered the ranges so fewer teachers would be at risk of receiving a low rating) at least once during the negotiations to accommodate union concerns,” the report says.

Morgaen Donaldson, a professor at the University of Connecticut, said the initial buy-in followed by pushback isn’t surprising, pointing to her own research in New Haven.

To some, aspects of the initiative “might be worth endorsing at an abstract level,” she said. “But then when the rubber hit the road … people started to resist.”

More effective teachers weren’t more likely to stay teaching, but less effective teachers were more likely to leave.

The basic theory of action of evaluation changes is to get more effective teachers into the classroom and then stay there, while getting less effective ones out or helping them improve.

The Gates research found that the new initiatives didn’t get top teachers to stick around any longer. But there was some evidence that the changes made lower-rated teachers more likely to leave. Less than 1 percent of teachers were formally dismissed from the places where data was available.

After the grants ran out, districts scrapped some of the changes but kept a few others.

One key test of success for any foundation initiative is whether it is politically and financially sustainable after the external funds run out. Here, the results are mixed.

Both Pittsburgh and Hillsborough have ended high-profile aspects of their program: the merit pay system and bringing in peer evaluators, respectively.

But other aspects of the initiative have been maintained, according to the study, including the use of classroom observation rubrics, evaluations that use multiple metrics, and certain career-ladder opportunities.

Donaldson said she was surprised that the peer evaluators didn’t go over well in Hillsborough. Teachers unions have long promoted peer-based evaluation, but district officials said that a few evaluators who were rude or hostile soured many teachers on the concept.

“It just underscores that any reform relies on people — no matter how well it’s structured, no matter how well it’s designed,” she said.

Correction: A previous version of this story stated that about half of the money for the initiative came from the Gates Foundation; in fact, the foundation’s share was 37 percent or about a third of the total.

reading science

Reading instruction is big news these days. Teachers, share your thoughts with us!

PHOTO: Helen H. Richardson, The Denver Post

Lately, lots of people are talking about reading. Specifically, how it’s taught (or not) in America’s schools.

Much of the credit is due to American Public Media reporter Emily Hanford. In September, she took an in-depth look at what’s wrong with reading instruction in the nation’s classrooms and how explicit, systematic phonics instruction could help.

The crux of the issue is this: In the 1980s and 1990s, the “whole language” approach to teaching reading took hold, relying on the idea that learning to read is a natural process that could be helped along by surrounding kids with good books. At many schools, phonics was out.

In time, many educators brought small doses of phonics back into their lessons, adopting an approach called “balanced literacy.” The problem is, neither whole language nor balanced literacy is based on science, Hanford explained.

Her work on the subject — an audio documentary called Hard Words, a follow-up Q&A for parents, and an opinion piece in the New York Times — has spawned much discussion on social media and elsewhere.

A Maine educator explained in her piece for the Hechinger Report why she agrees that explicit phonics instruction is important but doesn’t think “balanced literacy” should be thrown out. A Minnesota reporter examined the divide in her state over how much phonics should be included in reading lessons and how it should be delivered.

In a roundtable discussion on reading last spring, Stephanie Finn, a literacy coach in the West Genesee Central School District in upstate New York, described the moment she became disillusioned with the whole language approach. It was while reading a story with her young daughter.

“The story was about gymnastics and she had a lot of background knowledge about gymnastics. She loved gymnastics. She knew the word ‘gymnastics,’ and ‘balance beam’ and ‘flexible’ and she got to the girl’s name and the girl’s name was Kate, and she didn’t know what to do,” said Finn. “I thought ‘Holy cow, she cannot decode this simple word. We have a problem.’”

In an opinion piece in Education Week, Susan Pimentel, co-founder of StandardsWork, provides three recommendations to help educators promote reading proficiency. Besides not confining kids to “just-right” books where they already know most words, she says teachers should increase students’ access to knowledge-building subjects like science and social studies. Finally, she writes, “Let quality English/language arts curriculum do some of the heavy lifting. Poor-quality curriculum is at the root of reading problems in many schools.”

Meanwhile, some current and former educators are asking teacher prep program leaders to explain the dearth of science-based lessons on reading instruction.

An Arkansas teacher wrote in a letter to her former dean on Facebook, “while I feel like most of my teacher preparation was very good, I can say I was totally unprepared to teach reading, especially to the struggling readers that I had at the beginning of my career in my resource classroom.”

Former elementary school teacher Robert Pondiscio, a senior fellow and vice president for external affairs at the Thomas B. Fordham Institute, wrote to his former dean, “I’m grateful for the professional credential … But if there’s anything one might expect an advanced degree in elementary education to include, it would be teaching reading. It wasn’t part of my program.”

Teachers, now we’d like to hear from you. What resonates with you about the recent news coverage on reading instruction? What doesn’t? Share your perspective by filling out this brief survey.

Indiana's 2019 legislative session

Raising teacher pay likely to be at the forefront for Indiana lawmakers and advocates in 2019

PHOTO: Melanie Asmar/Chalkbeat
Colorado teachers rallied for more education funding on April 27, 2018.

Indiana lawmakers and education advocates are making raises for teachers a priority for the upcoming legislative session.

As top lawmakers — Republicans and Democrats — prepare to craft the next two-year state budget, they have been in talks about how money could be set aside for teachers and other educators. But it’s unclear how much of a pay hike is on the table or how the dollars would get from the state to teacher paychecks.

“The governor’s office and both Republican caucuses are seriously looking at this as an issue,” said Rep. Bob Behning, a Republican who chairs the House Education Committee. “If we’re focused on really making (teaching) more of a profession, you can’t do it by grants here, grants there. People need to see the opportunity.”

While Indiana’s teacher pay has not fallen as dramatically as it has in other states, salaries are down from 2009 when adjusted for inflation. The average teacher salary in 2018 was $54,846, down about 4.5 percentage points from nine years earlier, according to data from the National Education Association teachers union. An analysis of data from the National Center for Education Statistics and Council of Community and Economic Research ranks Indiana 18th highest in the nation for teacher salaries adjusted for cost of living.

Teacher pay has been central to education policy debates in 2018 across the nation, with teachers in several cities staging walkouts and protests to urge officials in their states to increase funding for classrooms. Indiana teachers have not gone on strike, but the national uproar around funding and teacher compensation has been felt among Hoosier educators — especially as schools across the state struggle to hire enough qualified teachers. In Indianapolis Public Schools, raising teacher pay was the driving motivation behind asking voters to approve a tax increase of $220 million over eight years.

“I don’t think I’ve talked to anyone who said they’re fully staffed in special education,” said Teresa Meredith, president of the Indiana State Teachers Association. “But if you get them and you can’t keep them because they can’t pay bills, and they have no hope of having a family or getting a house … they’re going to look elsewhere.”

It’s too early to know how lawmakers would approach raises logistically for the state’s more than 71,000 public school teachers or how much they’re willing to support, but there does seem to be some initial consensus that the increases should go to base salaries, not just stipends as previous efforts have involved.

“We need to look at how do we make a significant impact to the base for all teachers,” said Sen. Eddie Melton, a Democrat from northwest Indiana. “That’s where we’re going now, to figure out what’s a sustainable method to fund this — not just for one or two years, but ongoing.”

In previous years, the state has set aside a few million dollars at a time for teacher bonuses or stipends for teaching advanced courses or subjects in shortage areas, such as science, math, and special education. The state’s pool for merit pay raises this year for teachers rated effective and highly effective is $30 million, amounting to typically small bumps for teachers.

But a noticeable raise for every teacher in the state would cost many millions of dollars, a considerable undertaking at a time when state revenue has been shrinking and competition among lawmakers and agencies to get a slice of state funding is high.

It’s also unclear if the money for raises would be figured into the state’s school funding formula or as a separate line item. It could be especially complicated because in Indiana, there are no common teacher pay guidelines. Each district or charter school creates its own pay scale, which often involves union negotiations as well.

Lawmakers and advocates alike say they expect this to be a top issue for the legislature. Still, any proposal to increase teacher pay would be competing with other issues — chief among them increasing funding for the Department of Child Services. Earlier this year, the resignation of the agency’s director set off a major review of its staffing and caseload, stretched further by the number of children needing services because of Indiana’s opioid crisis.

Teacher salaries could also square off against other education issues, such as school safety improvements and initiatives to increase class offerings in science and math.

In Indianapolis Public Schools, district officials have been stressing the need to increase teacher pay — a key lever to convincing voters to pass a property tax increase to raise an additional $220 million for the district over eight years. Superintendent Lewis Ferebee said he’s also been having conversations with lawmakers about potential ways that the state could address the problem.

“They appreciate the need to address the teacher shortage, and they understand it’s an issue not only impacting Indianapolis Public Schools but it’s also an issue that’s statewide,” Ferebee told Chalkbeat two weeks ago.

Teacher hiring has continued to be a struggle for districts across the state, a survey from an Indiana State University professor said. Of the 220 districts surveyed, 91 percent said they’d had trouble filling jobs, with special education, science, and math being the hardest to fill.

According to state data, Indiana issued licenses to 4,285 new teachers in 2018, down slightly from 5,016 in 2017 and 4,566 in 2016. A survey conducted by the Indiana Department of Education reported 88 percent of educators who responded were unsatisfied with their pay, and it was the reason most frequently given for leaving the teaching profession.

“Based on conversations with some lawmakers, based on what’s going on across the country, I think our lawmakers have seen there’s reform fatigue,” Meredith said. “Let the dust settle and figure out how we come back and demonstrate respect for teachers.”

In other states where lawmakers have approved statewide teacher pay raises, the process has differed. Oklahoma raised the salary floor for all teachers, with an average increase of $6,100 per year. The state budgeted more than $425 million for the salary increases, which are to be covered by new higher taxes on cigarettes, cigars, and gas. In West Virginia, a nine-day strike ultimately led lawmakers to increase pay for all public employees by 5 percent.

Gov. Eric Holcomb has not yet weighed in on whether he would support a statewide teacher raise, but Behning said he’d been in conversations with the governor’s office. Indiana’s next legislative session begins in January.