Editor’s note: Kristin Klopfenstein is the executive director of the Education Innovation Institute at the University of Northern Colorado.
Several high-profile education reforms passed by the Colorado legislature in the last few years rely on massive collections of data to work as planned. For example, the 2009 accountability bill requires administrators at struggling schools to use school-level data to drive the improvement planning process.
Senate Bill 191’s teacher evaluation provisions require more, however. Administrators must be able to drill down to the individual level, accurately linking teachers with students to evaluate teachers based on how well their students progress over the year. And Senate Bill 10-036 tills the soil for teacher prep programs to monitor the achievement of their graduates’ students in order to improve teacher prep programs.
All are ambitious laws — and I sometimes fear that Colorado’s reform cart has raced ahead of the data horse. The Colorado Department of Education, school districts and the Department of Higher Education are still working out the details on the kinds of data needed. That’s not a criticism. Collecting data that links every student, teacher, school and public university in the state is incredibly slow and painstaking when done right – and you definitely want it done right. I just worry that the public enthusiasm for the reforms will fade before they even get a chance.
That would be a shame because Colorado is headed toward building one of the most sophisticated data systems in the country, one that can be used to help improve our schools in many ways. Administrators and teachers can use data to identify their schools’ weaknesses and work together to set targets and monitor improvement. Principals can provide useful feedback to individual teachers, helping the weakest improve or find a new profession. Researchers can measure which programs and reforms are most successful over time and examine why.
Collecting data that links every student, teacher, school and public university in the state is incredibly slow and painstaking when done right – and you definitely want it done right.Exemplary schools that do great things in this challenging fiscal environment can be identified and provide lessons for other schools. And policymakers and the public can see which schools are most effective and with which kinds of students.
Building a high quality statewide data system is a process that is near and dear to me. I am a Colorado native, and I just returned to the state to serve as executive director of the new Education Innovation Institute at the University of Northern Colorado. I spent much of the last decade working with the education database in Texas, which is one of the largest, most developed educational data systems in the country.
Our mission at EII is to bring researchers, practitioners and policymakers together to work on improving schools, an effort that demands top-notch data. One of our goals is to work with the state to help set standards for data and data collection to make sure Colorado comes up with a powerful and accessible data warehouse that serves all comers well.
Building complex data sets is key
The problem isn’t that schools and districts don’t know how to collect data; they’ve been reporting boatloads of it to comply with No Child Left Behind and other programs. Those data tell how districts, schools, and students perform on tests broken down by basic student demographic characteristics.
The big challenge now is to build datasets on individuals — students, teachers and student-teacher pairs — that collect much more information about what goes on during the year than just CSAP scores. Attendance rates, disciplinary offenses, courses taken, and grades earned can be combined to provide a valuable early warning system for students at risk of falling behind.
Money is tight, and effective data systems allow educators to apply resources efficiently to get the most bang for the buck. Once these data are linked over time, additional information emerges about whether students were promoted to the next grade, switched schools, graduated, dropped out, or went to college. Then we have the basis for determining how well Colorado public schools prepare kids for college and the workforce, as the 2008 CAP4Kids law mandates.
As an example of just how sticky data collection can get, consider the challenge of generating a set of links between students and their teachers. Without this, it is impossible for outsiders to tell which teachers are really making a difference and which are just coasting. Such links between students and their “teacher of record” are at the heart of new evaluation systems that base half a teacher’s annual appraisal on changes in her students’ test scores. But creating clear, fair links is not easy.
At first blush, it seems that schools always know which teachers students are with each hour of the day. And in middle and high schools, they usually do. Elementary schools, however, tend to informally move students about in ways that are educationally sound but often not documented by the main office.
For example, Mrs. Smith is a reading specialist while Mrs. Jones is an expert in elementary mathematics, so all the kids from Mrs. Smith’s class have Mrs. Jones’ for math and vice versa for reading. Then Mrs. Smith goes out on maternity leave, and Mr. Black substitutes for four months. Should Mrs. Smith get any credit for the students assigned to her in the school roster who score through the roof in math? If she incorporates math word problems into the reading curriculum, what then?
And how much credit should Mrs. Jones get for student scores in science if she team teaches that subject with Mrs. Smith? Whatever the solution, it is critical that the process be transparent and consistent. Consistency among districts is also important to discourage district-shopping among teachers seeking the most favorable approach.
Another issue is student mobility. Johnny starts school in the fall in Mrs. Smith’s class but moves to another school in the district part way through the year. How long did Mrs. Smith need to have him in class to be responsible for learning gains on the spring CSAP? Maybe Mrs. Smith has an aide who works with Johnny individually on his reading problems twice a week. Johnny is touched by Mrs. Smith and her aide, Mrs. Jones, Mr. Black, and all the people at his new school as well.
Another big challenge is to make sure every school and district across the state collects the same data and formats it in exactly the same way.An exasperated principal might be tempted not to assign Johnny to any single educator of record. However, Johnny is one of the students most in need of the attention accountability systems are designed to provide.
If a data system is well designed and fully fleshed out it can provide a wealth of information. It can help educators flag problems through the year rather than waiting until CSAP scores come out in the summer, which is important because students who are struggling in multiple subjects at mid-year are very likely to be retained in grade, as are students with attendance and discipline problems.
We should be using all available information
Teachers need to know how students are behaving in other teachers’ classrooms and how they behaved in earlier grades. All this information can be recorded for each student and made accessible to current teachers and administrators, even if a student previously attended school in a different Colorado district. As I said, this is important for keeping up with students who move a lot, a group that research shows is highly vulnerable to falling behind and dropping out.
Even with CSAP, we’re not using all the available information. Recent research by a team at CDE uses sixth-grade CSAP scores to predict whether a student is likely to need college remediation after high school graduation. We need to be using this information to intervene as soon as students start to struggle, rather than just watching as they fulfill the destiny of their current path.
Another big challenge is to make sure every school and district across the state collects the same data and formats it in exactly the same way. Today, only a handful of Colorado districts already have the capacity to effectively match students with one or more educators of record – and they probably developed their procedures without much consultation with the others.
To understand why this could be a problem, think of railroads before details like track gauges were standardized. If all districts use different data systems, we don’t have an effective way to follow students who move around, even within Colorado. This means a student who was getting great help in one district might lose valuable momentum if he moves to another. Just like with the railroad, it is important to recognize when the benefits of coordinated data systems outweigh the costs in terms of local control.
State agencies (CDE and DHE) have made tremendous progress in a very short time with limited resources to develop a data infrastructure. They still have a ways to go, but their progress has been strong and steady. It is incumbent on policymakers, practitioners, and the public to recognize that it will take time to implement this bundle of reforms right. The key to efficiency is good information and the returns to a 21st century administrative data system will pay off for decades to come.
About our First Person series:
First Person is where Chalkbeat features personal essays by educators, students, parents, and others trying to improve public education. Read our submission guidelines here.