Scott Alexander has an exceptional post on teacher effectiveness research. You should read it.
He concludes with:
In summary: teacher quality probably explains 10% of the variation in same-year test scores, which corresponds to a +1 SD better teacher causing a +0.1 SD student test score improvement, which isn’t that much.
This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects.
…and most people agree that these sorts of scores combine some signal with a lot of noise.
For some reason, even though teachers’ effects on test scores decay very quickly, studies have shown that they have significant impact on earning as much as 20 or 25 years later, so much so that kindergarten teacher quality can predict thousands of dollars of difference in adult income.
This seemingly unbelievable finding has been replicated in quasi-experiments and even in real experiments and is difficult to banish. Since it does not happen through standardized test scores, the most likely explanation is that it involves non-cognitive factors like behavior.
I really don’t know whether to believe this and right now I say 50-50 odds that this is a real effect or not – mostly based on low priors rather than on any weakness of the studies themselves. I don’t understand this field very well and place low confidence in anything I have to say about it.
All of this further pushes me down the road that the VAM / teacher evaluation wars have been a waste of political and financial capital.
I’ve always believed that choice and charter were a better bet, and that is where I’ve spent a 100% of my energy and philanthropic capital, but I hadn’t quite internalized the brutal math of teacher reforms: that +1SD teacher only leads to a +.1SD test score improvement.
Given how much ink is spilled on the teacher wars, I don’t blame Scott for conducting a teacher effectiveness review.
But what Scott failed to cover in-depth is what we know about effective school and systems level reforms.
See below for some illustrative data about why school effects paint a much different picture than teacher effects [note: I’m not as good at statistics as Scott, so please let me know where I’ve made analytical errors].
See below for the effects of attending an urban charter school over time:
If a student spends four years in a charter, she gets a +.1-.15 bump in test scores.
And here’s the kicker: we’re not talking about the best charter schools in the nation; this is the average effect of urban charter schools across 41 cities.
Moreover, unlike the decay you see in teacher effectiveness research, the effect of the charter impact actually increases over time.
So if a student (especially a low-income students) wants a +.1 bump in test scores, she can either:
(1) search across the city for a teacher who is +1 SD above the average teacher (and pray that her next few teachers are also good so the effect doesn’t decay), or
(2) she can just enroll in an average urban charter school for a few years.
Here’s another thing: urban charter schools are quickly getting better over time.
The average impact on urban charter has nearly doubled between 2009 and 2012.
The reason for this is most likely simple: entrepreneurship, choice based competition, and evolution are leading to rapid performance improvements even as the sector continues to scale.
Also, we have emerging evidence that these impacts can scale across an entire city.
As New Orleans progressed to 100% charter, it saw achievement gains of about +.4 SD over five years.
Moreover, these gains that were marked by continuous improvement, not fadeout.
Of course, the New Orleans reforms happened under very unique circumstances, so we’re not yet sure if we can replicate this at scale, but the CREDO urban charter data does give us reason to believe that urban charters can have positive effects at scale across the country.
Now, test scores aren’t everything.
For one, test scores gains often reflect gains in crystallized rather than fluid intelligence:
Additionally, even charter sectors with mediocre test gains, such as those in Florida, seem to be adding value:
So while test scores give us some information, they clearly don’t give us all the information we need, which is why I believe deeply in parental choice, as well as only intervening in schools with poor test scores when their growth scores are in the very bottom of a city or state.
It takes an exceptional teacher to marginally increase a student’s test score, and these gains fade out quickly.
It takes an average urban charter school to increase a student’s test score, and these gains increase over time.
Lastly, test scores aren’t everything, so we should be cautious in how we use them and we should give strong deference to parental choice.
Did I get this right? Did I make any big methodological errors?