Missing the Schools for the Teachers (the Folly of the Teacher Wars)

Scott Alexander has an exceptional post on teacher effectiveness research. You should read it.

He concludes with:

In summary: teacher quality probably explains 10% of the variation in same-year test scores, which corresponds to a +1 SD better teacher causing a +0.1 SD student test score improvement, which isn’t that much.

This decays quickly with time and is probably disappears entirely after four or five years, though there may also be small lingering effects.

…and most people agree that these sorts of scores combine some signal with a lot of noise.

For some reason, even though teachers’ effects on test scores decay very quickly, studies have shown that they have significant impact on earning as much as 20 or 25 years later, so much so that kindergarten teacher quality can predict thousands of dollars of difference in adult income.

This seemingly unbelievable finding has been replicated in quasi-experiments and even in real experiments and is difficult to banish. Since it does not happen through standardized test scores, the most likely explanation is that it involves non-cognitive factors like behavior.

I really don’t know whether to believe this and right now I say 50-50 odds that this is a real effect or not – mostly based on low priors rather than on any weakness of the studies themselves. I don’t understand this field very well and place low confidence in anything I have to say about it.

All of this further pushes me down the road that the VAM / teacher evaluation wars have been a waste of political and financial capital.

I’ve always believed that choice and charter were a better bet, and that is where I’ve spent a 100% of my energy and philanthropic capital, but I hadn’t quite internalized the brutal math of teacher reforms: that +1SD teacher only leads to a +.1SD test score improvement.

____

Given how much ink is spilled on the teacher wars, I don’t blame Scott for conducting a teacher effectiveness review.

But what Scott failed to cover in-depth is what we know about effective school and systems level reforms.

See below for some illustrative data about why school effects paint a much different picture than teacher effects [note: I’m not as good at statistics as Scott, so please let me know where I’ve made analytical errors].

___

See below for the effects of attending an urban charter school over time:

Screen Shot 2016-05-21 at 5.43.09 PM

If a student spends four years in a charter, she gets a +.1-.15 bump in test scores.

And here’s the kicker: we’re not talking about the best charter schools in the nation; this is the average effect of urban charter schools across 41 cities.

Moreover, unlike the decay you see in teacher effectiveness research, the effect of the charter impact actually increases over time.

So if a student (especially a low-income students) wants a +.1 bump in test scores, she can either:

(1) search across the city for a teacher who is +1 SD above the average teacher (and pray that her next few teachers are also good so the effect doesn’t decay), or

(2) she can just enroll in an average urban charter school for a few years.

___

Here’s another thing: urban charter schools are quickly getting better over time.

Screen Shot 2016-05-21 at 5.53.52 PMThe average impact on urban charter has nearly doubled between 2009 and 2012.

The reason for this is most likely simple: entrepreneurship, choice based competition, and evolution are leading to rapid performance improvements even as the sector continues to scale.

___

Also, we have emerging evidence that these impacts can scale across an entire city.

Screen Shot 2015-08-04 at 7.03.28 AM

As New Orleans progressed to 100% charter, it saw achievement gains of about +.4 SD over five years.

Moreover, these gains that were marked by continuous improvement, not fadeout.

Of course, the New Orleans reforms happened under very unique circumstances, so we’re not yet sure if we can replicate this at scale, but the CREDO urban charter data does give us reason to believe that urban charters can have positive effects at scale across the country.

___

Now, test scores aren’t everything.

For one, test scores gains often reflect gains in crystallized rather than fluid intelligence:

ednext_XIV_4_west_fig01-small

Additionally, even charter sectors with mediocre test gains, such as those in Florida, seem to be adding value:

florida

So while test scores give us some information, they clearly don’t give us all the information we need, which is why I believe deeply in parental choice, as well as only intervening in schools with poor test scores when their growth scores are in the very bottom of a city or state.

___

In summary:

It takes an exceptional teacher to marginally increase a student’s test score, and these gains fade out quickly.

It takes an average urban charter school to increase a student’s test score, and these gains increase over time.

Lastly, test scores aren’t everything, so we should be cautious in how we use them and we should give strong deference to parental choice.

Did I get this right? Did I make any big methodological errors?

10 thoughts on “Missing the Schools for the Teachers (the Folly of the Teacher Wars)

  1. Robert Pondiscio

    <<< All of this further pushes me down the road that the VAM / teacher evaluation wars have been a waste of political and financial capital.

    Yup. Not the hill on which I want to die. Never has been. So many more important fights to be waged.

    Like

    Reply
  2. Mike

    Hi Neerav,

    Enjoyed that Alexander post too, and yours:

    You don’t comment on the interaction of teacher quality and charter gains.

    I.e., either

    a. The charter attracts, on average, more higher performing teachers, or sheds more low-performing teachers

    b. The charter raises all boats: puts all teachers in better position to succeed

    c. Something is happening at the charter that doesn’t have much to do with the teachers

    Not to defend the districts in the “how” – but to extent you believe “a” is going on in the charters, isn’t VAM their (perhaps ill-fated) effort to do the same?

    Like

    Reply
    1. nkingsl Post author

      I think (b) is a big enough effect that I wouldn’t bet the farm on (a). Also, teacher eval stuff has yet to attract / remove teachers at rate charters tend to do… so it seems like a pretty ineffective way (so far) to accomplish (a)

      Like

      Reply
  3. Ciro Curbelo

    Individuals working in complex human systems can only be as good as the systems within which they operate. An average cook will do a better job working at a Cheesecake Factory than they will working at your run of the mill diner. An average surgeon working at Columbia Presbyterian will likely have better outcomes than they would working at a run of the mill local community hospital. Conversely, rock stars working in crappy systems can’t really excel past the crappy systems. How many of us have suffered working for bad bosses or mediocre organizations only to turn it around after switching bosses and companies? Teacher eval proponents are disingenuous when they ignore the systems teachers work within. And by starting with teacher eval, they have only made those systems more resistant to change. The best hope for career teachers who want to excel is to find higher functioning systems to join – or to create them. Same goes with any professional – if you want to be happy and fulfilled day to day, find an organization that makes you better.

    Liked by 1 person

    Reply
  4. Jake

    Does this account for the self-selection effects associated with charter schools? Most of what I have read about the issue says that charter schools are able to attract a higher quality of student in the first place, since their parents actually have to do something to get them enrolled in the charter school. This self selects for students with parents that actually care, even a little bit, which in itself could explain a lot of the gains from charter schools. I still love the idea, since if it works, there should be more of it, but I’m not sure it provides a fully generalizable framework to base all schools on.

    Like

    Reply
    1. nkingsl Post author

      CREDO controls for 10 or so variables, including poverty and race – but it’s all quasi-experimental so surely could be selection effects over time…

      Like

      Reply
    2. deliveratormatt

      Yeah, and charters specifically weed out the *worst* behaving students. See the original slatestarcodex post for some extensive reasoning about that. (As a teacher, I’ve definitely been in situations where, if I could have permanently kicked out the worst-behaved 1-2 students in a group, I could have easily made the entire rest of the class !20% more productive.) Part of that weeding, in fact, is getting rid of or not accepting students with various disabilities, for which charters are rightly beginning to get in serious legal trouble.

      They also work their teaching forces practically to death, which is not exactly a sustainable model. That’s starting to change in a few places, but not enough. And not just for moral reasons!

      Let’s say that charter schools do, for whatever reason, create and/or attract better teachers. If they’re driving all those people out of the profession—and evidence says they are—how can that *possibly* be a good thing in the long run?

      Like

      Reply
  5. Horace Manifesto (@horacemanifesto)

    Really important analysis. I’d add that an important factor is that charter networks are extending their functions into teacher pre-service (through fellowships), higher ed (through Relay etc) and intensive in-service development. This allows them to reduce variations in teacher quality, meaning that kids get successive better-than-average teachers, which dramatically increases the teacher effect. My fear is that this human capital structure is reaching a ceiling. It depends on teachers putting in a ton of semi-compensated time (60 hours weeks at $50k?) – in effect, subsidizing the model. When you’ve got a deep pipeline of overachieving, upper middle class recent (which means generally white) grads who have the motivation and financial wherewithal to do this, it more or less works. But there are signs that that pipeline is starting to shrink, and the hours:pay ratio starts to become personally unsustainable as you approach 30, which is also when teachers start to reach peak effectiveness. It’s fine to say that an NYC charter teacher with 5 years of experience receives 10-20% more pay than they would in the district, but that doesn’t mean the salary is enough, in an absolute sense, to save for retirement or pay for childcare. As the big networks increasingly recognize the importance of deeper learning that builds fluid intelligence, they’re confronting this reality but I haven’t seen a compelling model emerge yet. Fundamentally, districts manage this tension by 1) backloading compensation with pensions, and 2) limiting hours worked. To me, the bottom line is that to have any effective systemic human capital approach, you need a lot of extra hours from teachers so that they can develop individually and align organizationally. Charters have been getting these hours without paying market rates for them, and the stress is starting to show. Recruitment is getting tougher, retention has always been a challenge, and some surprising talent gaps are starting to show through.

    Like

    Reply
    1. nkingsl Post author

      Good points. It will be interesting to see how charter human capitals adjust as the sector scales. My gut is that internal residencies plus better instructional systems will be key.

      Like

      Reply
  6. Pingback: First blog post | grayemptiness

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s