a high-stakes evaluation

The Gates Foundation bet big on teacher evaluation. The report it commissioned explains how those efforts fell short.

PHOTO: Brandon Dill/The Commercial Appeal
Sixth-grade teacher James Johnson leads his students in a gameshow-style lesson on energy at Chickasaw Middle School in 2014 in Shelby County. The district was one of three that received a grant from the Gates Foundation to overhaul teacher evaluation.

Barack Obama’s 2012 State of the Union address reflected the heady moment in education. “We know a good teacher can increase the lifetime income of a classroom by over $250,000,” he said. “A great teacher can offer an escape from poverty to the child who dreams beyond his circumstance.”

Bad teachers were the problem; good teachers were the solution. It was a simplified binary, but the idea and the research it drew on had spurred policy changes across the country, including a spate of laws establishing new evaluation systems designed to reward top teachers and help weed out low performers.

Behind that effort was the Bill and Melinda Gates Foundation, which backed research and advocacy that ultimately shaped these changes.

It also funded the efforts themselves, specifically in several large school districts and charter networks open to changing how teachers were hired, trained, evaluated, and paid. Now, new research commissioned by the Gates Foundation finds scant evidence that those changes accomplished what they were meant to: improve teacher quality or boost student learning.  

The 500-plus page report by the Rand Corporation, released Thursday, details the political and technical challenges of putting complex new systems in place and the steep cost — $575 million — of doing so.

The post-mortem will likely serve as validation to the foundation’s critics, who have long complained about Gates’ heavy influence on education policy and what they call its top-down approach.

The report also comes as the foundation has shifted its priorities away from teacher evaluation and toward other issues, including improving curriculum.

“We have taken these lessons to heart, and they are reflected in the work that we’re doing moving forward,” the Gates Foundation’s Allan Golston said in a statement.

The initiative did not lead to clear gains in student learning.

At the three districts and four California-based charter school networks that took part of the Gates initiative — Pittsburgh; Shelby County (Memphis), Tennessee; Hillsborough County, Florida; and the Alliance-College Ready, Aspire, Green Dot, and Partnerships to Uplift Communities networks — results were spotty. The trends over time didn’t look much better than similar schools in the same state.

Several years into the initiative, there was evidence that it was helping high school reading in Pittsburgh and at the charter networks, but hurting elementary and middle school math in Memphis and among the charters. In most cases there were no clear effects, good or bad. There was also no consistent pattern of results over time.

A complicating factor here is that the comparison schools may also have been changing their teacher evaluations, as the study spanned from 2010 to 2015, when many states passed laws putting in place tougher evaluations and weakening tenure.

There were also lots of other changes going on in the districts and states — like the adoption of Common Core standards, changes in state tests, the expansion of school choice — making it hard to isolate cause and effect. Studies in Chicago, Cincinnati, and Washington D.C. have found that evaluation changes had more positive effects.

Matt Kraft, a professor at Brown who has extensively studied teacher evaluation efforts, said the disappointing results in the latest research couldn’t simply be chalked up to a messy rollout.

These “districts were very well poised to have high-quality implementation,” he said. “That speaks to the actual package of reforms being limited in its potential.”

Principals were generally positive about the changes, but teachers had more complicated views.

From Pittsburgh to Tampa, Florida, the vast majority of principals agreed at least somewhat that “in the long run, students will benefit from the teacher-evaluation system.”

Source: RAND Corporation

Teachers in district schools were far less confident.

When the initiative started, a majority of teachers in all three districts tended to agree with the sentiment. But several years later, support had dipped substantially. This may have reflected dissatisfaction with the previous system — the researchers note that “many veteran [Pittsburgh] teachers we interviewed reported that their principals had never observed them” — and growing disillusionment with the new one.

Majorities of teachers in all locations reported that they had received useful feedback from their classroom observations and changed their habits as a result.

At the same time, teachers in the three districts were highly skeptical that the evaluation system was fair — or that it made sense to attach high-stakes consequences to the results.

The initiative didn’t help ensure that poor students of color had more access to effective teachers.

Part of the impetus for evaluation reform was the idea, backed by some research, that black and Hispanic students from low-income families were more likely to have lower-quality teachers.  

But the initiative didn’t seem to make a difference. In Hillsborough County, inequity expanded. (Surprisingly, before the changes began, the study found that low-income kids of color actually had similar or slightly more effective teachers than other students in Pittsburgh, Hillsborough County, and Shelby County.)

Districts put in place modest bonuses to get top teachers to switch schools, but the evaluation system itself may have been a deterrent.

“Central-office staff in [Hillsborough County] reported that teachers were reluctant to transfer to high-need schools despite the cash incentive and extra support because they believed that obtaining a good VAM score would be difficult at a high-need school,” the report says.

Evaluation was costly — both in terms of time and money.

The total direct cost of all aspects of the program, across several years in the three districts and four charter networks, was $575 million.

That amounts to between 1.5 and 6.5 percent of district or network budgets, or a few hundred dollars per student per year. Over a third of that money came from the Gates Foundation.

The study also quantifies the strain of the new evaluations on school leaders’ and teachers’ time as costing upwards of $200 per student, nearly doubling the the price tag in some districts.

Teachers tended to get high marks on the evaluation system.

Before the new evaluation systems were put in place, the vast majority of teachers got high ratings. That hasn’t changed much, according to this study, which is consistent with national research.

In Pittsburgh, in the initial two years, when evaluations had low stakes, a substantial number of teachers got low marks. That drew objections from the union.

“According to central-office staff, the district adjusted the proposed performance ranges (i.e., lowered the ranges so fewer teachers would be at risk of receiving a low rating) at least once during the negotiations to accommodate union concerns,” the report says.

Morgaen Donaldson, a professor at the University of Connecticut, said the initial buy-in followed by pushback isn’t surprising, pointing to her own research in New Haven.

To some, aspects of the initiative “might be worth endorsing at an abstract level,” she said. “But then when the rubber hit the road … people started to resist.”

More effective teachers weren’t more likely to stay teaching, but less effective teachers were more likely to leave.

The basic theory of action of evaluation changes is to get more effective teachers into the classroom and then stay there, while getting less effective ones out or helping them improve.

The Gates research found that the new initiatives didn’t get top teachers to stick around any longer. But there was some evidence that the changes made lower-rated teachers more likely to leave. Less than 1 percent of teachers were formally dismissed from the places where data was available.

After the grants ran out, districts scrapped some of the changes but kept a few others.

One key test of success for any foundation initiative is whether it is politically and financially sustainable after the external funds run out. Here, the results are mixed.

Both Pittsburgh and Hillsborough have ended high-profile aspects of their program: the merit pay system and bringing in peer evaluators, respectively.

But other aspects of the initiative have been maintained, according to the study, including the use of classroom observation rubrics, evaluations that use multiple metrics, and certain career-ladder opportunities.

Donaldson said she was surprised that the peer evaluators didn’t go over well in Hillsborough. Teachers unions have long promoted peer-based evaluation, but district officials said that a few evaluators who were rude or hostile soured many teachers on the concept.

“It just underscores that any reform relies on people — no matter how well it’s structured, no matter how well it’s designed,” she said.

Correction: A previous version of this story stated that about half of the money for the initiative came from the Gates Foundation; in fact, the foundation’s share was 37 percent or about a third of the total.

First Person

We work at Denver’s Title I schools, too. Here’s why we’re ready to strike.

PHOTO: AAron Ontiveroz/The Denver Post
Students in kindergarten on the first day of school at McGlone Academy.

We are a group of teachers representing schools in the far northeast region of Denver. Many of us now receive “incentives” for working at Title I schools where many students live in poverty — and we are also willing to strike in support of the union’s proposed salary structure, which moves some of the money used for those incentives into long-term base pay.

Why? In short, we would rather have our base pay prioritized than earn bonuses that are not reliable, may not be working, and may also take the pressure off the district to solve systemic problems our schools face.

Issue #1: The current bonuses can’t be relied on. The “hard to serve” school label is based on free and reduced-price lunch percentages, which vary on an annual basis. Teachers at Marie L. Greenwood Academy, for example, could lose their “hard to serve” label because their school dropped just barely below the threshold. Additionally, John Amesse Elementary School and McGlone Academy are less than two miles apart, serve similar populations, and are a part of the same network; however, due to ambiguous calculations based on test scores, free and reduced-price lunch numbers, and teacher turnover rates, McGlone teachers receive larger bonuses than John Amesse teachers. This is not fair nor equitable. Teachers need money they can depend on.

Issue #2: It’s not clear that the current bonuses are working. We have not seen conclusive evidence that the incentives we receive for working in hard to serve schools have affected teacher retention or recruitment. Every year, schools in our area are hiring for positions that often get filled by first-year teachers. Many of the schools that receive these incentives still suffer from the same high turnover rates the bonuses were meant to remedy.

Issue #3: The current bonuses let the district off the hook. Some have argued that teachers in Title I schools deserve significant bonuses because the challenges faced in our work are difficult and taxing. However, many of these issues are due to systemic problems that the district would be better off trying to solve directly.

We know that increasing incentive pay to work at “hard to serve” schools will not fix the issues around segregation in Denver Public Schools. Increasing incentive pay to work at “hard to serve” schools will not fix the issues around some schools lacking nurses, social workers, counselors, support for Spanish speaking and emerging bilingual students, and support for special education programs. It will not solve issues around the lack of reliable technology, funding for arts, comprehensive neighborhood schools, or the flood of issues that we all feel in our schools on a daily basis.

We support the union’s proposal because we want the decisions we make as educators to stem from a love of our schools, a desire to serve our students, and a hope to support our community. We want teachers to seek out and stay at our schools because they believe in our vision, our mission, our students, and our community.

We are also passionate about a clear and transparent pay schedule. We want that structure to recognize our dedication to the field and our commitment to furthering our education – not a system that provides one-time bonuses that are in our checks one year and absent the next due to circumstances outside our control.

Anyone who enters our classrooms will see that we are doing our best with the resources we have in order to lift up the students in Denver who are most impacted by systemic racism and poverty. Let us come together on this idea: Fair pay for teachers means better outcomes for students. If we can stand together on this, then we can help improve the lives of so many more students, teachers, and families.

This piece was written by Jessica Schneider, Noel Community Arts School; Tanessa Bass, John H. Amesse Elementary; Rebecca Roberts, John H. Amesse Elementary; Valerie Henderson, Sandra Todd Williams Academy; Brian Weaver, Florida Pitt Waller ECE-8; Michelle Garrison, Farrell B. Howell ECE-8; Michael Sitkin, DCIS @ Montbello; Cory Montrieul, DCIS @ Montbello; and Nik Arnoldi, Escalante-Biggs Academy.

Last minute

Teachers union continues voting on possible Denver strike

PHOTO: Michael Ciaglo/Special to the Denver Post
Eagleton Elementary School first grade teacher Valerie Lovato, left, and East High School French teacher Tiffany Choi hold up signs as the Denver teachers union negotiates with district officials.

Denver teachers resumed voting Tuesday evening on whether to go on strike, a decision that will touch tens of thousands of people in Colorado’s largest school district.

The vote comes after months of negotiations left Denver Public Schools and the Denver Classroom Teachers Association still $8 million apart and with serious philosophical disagreements about how teacher compensation should be structured. Denver teachers are riding a wave of activism by their peers across the country that began last year and continues to build. Teachers in Los Angeles just reached a tentative deal with their district after more than a week on strike.

Members of the teachers union began voting on a strike Saturday. A final round of voting began at 4 p.m. Tuesday and will end at 9 p.m. Union officials said Tuesday evening that results would be announced at 9:30 p.m.  

On Tuesday evening, a steady stream of teachers bundled against the cold made their way into a Knights of Columbus Hall in downtown Denver where the last voting session is taking place.

Maria Cruz, an early childhood education teacher for the past two years who previously worked as a paraprofessional in the district, said she voted “yes” to strike hoping it will push the district to close the gap between its offer and what the union is seeking.  

“Teachers come and go and come and go and they never stay because there is not enough pay,” she said. “It doesn’t validate the teaching profession.”

The earliest a strike could start would be Jan. 28. On Tuesday evening, district families received a robocall from Denver Superintendent Susana Cordova making clear that classes would go on as normal on Wednesday, and that district officials intend to keep schools open for the foreseeable future.   

Cordova has said she’ll ask for state intervention if the vote is yes, which could delay a strike. If teachers do walk out, the district intends to keep schools open and students learning by relying on substitutes, tapping central office staff with past teaching experience, and using pre-packaged lessons plans for every grade and subject area. 

A Denver strike would affect roughly 71,000 students in district-run schools.

District officials went on the offensive over the weekend, making the argument that their offer was generous and responsive to longstanding teacher complaints about stagnant salaries.   

The district also published its new salary schedule online alongside the salary schedules of other Denver metro area districts.

The two sides disagree on how much new money the district should put into teacher compensation and also on how that compensation should be structured. The district has said it will not compromise on offering bonuses to teachers at high-poverty and hard-to-serve schools. The union wants smaller bonuses and more money to go into base pay.

This would be the first teacher strike in Denver in 25 years.