Chicago’s Black educators who taught in high-poverty schools were more likely to receive lower scores on teacher evaluations, new research shows, pointing to bias in a widely used measure that determines promotion and tenure.
Left unadjusted, the scores could lead to disproportionate and incorrect identification of Black teachers for remediation and dismissal — an outcome that would be counterproductive to efforts to diversify the teacher workforce. About half of Chicago teachers are white.
The study, which examined two years of classroom observation data from 2013 to 2015, published Thursday in American Educational Research Association’s peer-reviewed journal “Educational Evaluation and Policy Analysis.” Classroom observation scores make up the bulk of a teacher’s evaluation under Chicago’s REACH system, which separate studies have linked to measurable improvements in student achievement.
Now researchers have found that some adjustments may be needed.
The study by Matthew P. Steinberg, an associate professor at George Mason University, and Lauren Sartain, formerly of the University of Chicago Consortium on School Research and currently at the University of North Carolina at Chapel Hill, found that the average Black teacher in Chicago ranked at the 37th percentile in classroom observation scores, compared to the 55th percentile for the typical white teacher.
But there was another critical consideration: Those Black educators also were more likely to teach in the city’s most economically disadvantaged schools.
Once researchers adjusted for certain factors — such as the percentage of a teacher’s students from low-income families, the students’ prior year test scores and behavior records, and even school-level resources — that gap statistically disappeared.
“It’s not because Black teachers are less effective or have lower performance than white teachers in Chicago. It’s that Black teachers in Chicago teach systematically in very different school settings,” said Steinberg. “Black teachers are systematically in schools that are less resourced. It’s not just the classroom environment, it’s the package of goods and services at the school level that is explaining differences in observation scores.”
Researchers said Black teachers also tended to teach in schools with less consistent leadership and less support for ambitious instruction. The race of the evaluator did not appear to affect the score.
Steinberg said the findings have national implications. Emerging evidence from Boston, Michigan, and other places shows significant gaps between Black and white teachers when it comes to classroom observation scores.
“We think these findings are important for districts nationwide,” he said.
The good news: Districts can make statistical adjustments to try to control for differences across schools and classrooms.
“Teachers and principals generally like the observations and use the information from those observations to make improvements to their practice. But clearly as we show in this paper, there are issues as well,” said Sartain, who has previously studied the REACH system. “Policymakers can now make decisions about how to strengthen the evaluation system with this information in hand.”
James Gherardi, a spokesman for Chicago Public Schools, said it was a district priority to provide an evaluation system that is “fair, equitable and drives school improvement” and the district had commissioned a joint study with the Chicago Teachers Union to fine-tune the system and better prevent bias.
“The district remains committed to working with our research and labor partners to refine the tools used to provide teacher feedback and improve the quality of instruction our students receive,” Gherardi said.
The study examined scores for more than 5,500 teachers from 411 CPS elementary schools. The teachers taught in grades kindergarten through fifth grade.
A spate of recent research links teachers of color to better outcomes for students of color. But across the country, and in Chicago, retention of Black educators has been a point of concern.
According to a Chalkbeat analysis, by 2019, Chicago had lost a quarter of its Black teaching force over a six-year period. At the same time, more than 9 in 10 white teachers remained in the district, and the number of Latino teachers grew.
Union officials previously have pointed to layoffs of large numbers of Black teachers and filed litigation against the district over the issue.
In more recent years, Chicago officials have made investments to better support, recruit and retain staff, particularly in high-need schools.
Chicago’s adoption of the REACH system made the city a test case for how to revamp teacher evaluations. The system, piloted in 2012 and tweaked over time through a joint committee staffed by district leadership and the teachers union, was widely considered more comprehensive and effective than a previous checklist that the district had used since the 1970s.
In March 2020, researchers at the University of Chicago Consortium published a five-year look back at the system based largely on educator surveys. Most teachers and administrators reported that the evaluation system improved teacher practice and student learning.