My response to Freddie deBoer

Freddie deBoer recently commented on my post on educational productivity. Freddie is a rising star in the blogosphere, and his concerns mirror the concerns of many others, so I thought responding to his comment could be of use.

Below, I respond to his critiques. With Freddie’s permission, I’ve also pasted in his full comment at the end of the post. It might be helpful to start there.

I. Do Charters Outperform Traditional Schools? 

Freddie begins:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools.

This is true when it comes to all students. But is not true when it comes to disadvantaged students, especially those served by urban charter schools.

CREDO’s urban charter study found the following effects for urban charters:

Screen Shot 2015-03-18 at 7.39.20 PM

The study also found positive charter effects for disadvantaged populations:

Screen Shot 2017-06-15 at 8.51.23 AM

CREDO’s national study found similar positive effects for disadvantaged populations.

CREDO is the best national quasi-experimental data source we have, and its methodology holds up well in comparisons with experimental data.

To the extent you’re suspicious of CREDO or of quasi-experimental design, Rand also did a national study on charters that looked exclusively at experimental studies. The authors found:

“Consistent with many previous studies that have focused on broad sets of charter schools, we found no evidence that, on average, attending charter schools had a positive impact on student achievement. The estimated impact of attending the average charter school in the study was negative but not statistically signicant after adjusting for the multiple hypotheses tested. However, the average impact of attending charter schools in large urban areas or those serving lower achieving or more disadvantaged students was large and positive.”

In a NBER working paper, Dennis Epple, Richard Romano, and Ron Zimmer offer their summary of charter RCT research:

“These studies have been much more supportive of charter schools with nearly all of these studies finding positive effects—in some cases, quite large effects (Hoxby and Rockoff, 2004; Hoxby, Kang, & Murarka, 2009; Abdulkadiroglu, et al., 2010; Curto and Fryer, 2011; Tuttle, et al., 2013; Wong, et al., 2014)—with only one finding no effect, a study by Mathematica of charter middle schools (Gleason, et al., 2010).”

For what it’s worth, my reading of the Mathematica study (which the above authors say is the only finding with no effect), is that it does find positive math impacts in for disadvantaged students, see pages 70-71,78 in this report. 

Apple, Romano, and Zimmer also provide a good analysis of the pros and cons of quasi-expermintal studies and experimental studies, as well as the trickiness of solving for selection effects.

While none of the evidence is perfect, I think it’s very reasonable to hold the belief that charters are serving disadvantaged students in urban areas better than traditional public schools.

At the very least, I would not call this belief “a basic failed assumption.”

II. Can We Replicate the Success of New Orleans?

Freddie writes:

Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities. So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

I don’t know if the New Orleans efforts can be scaled, but I don’t think the idea is absurd.

Rather, I think we should see if the structural reforms of New Orleans can be applied with success in additional cities.

Very specifically, I think it would be great to get 8-10 cities where:

(1) A majority of the schools are non-profit managed.

(2) A unified enrollment system allows families to easily choose from a variety of schools.

(3) A unified accountability system provides parents information about school quality and leads to governance change in the lowest performing schools.

I view these as the core tenets of the New Orleans model.

Big picture, I don’t understand why we’d dismiss the incredible achievement gains in New Orleans rather than try to learn from them and see if they can work elsewhere.

As a reminder, the New Orleans achievement gains were very large:

Screen Shot 2015-08-04 at 7.03.28 AM

III. Can We Replicate the Success of High-Performing Charter Management Organizations? 

Freddie writes:

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs.
Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent. These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed.

I agree with Freddie that Success Academy might not be scalable outside of New York City (or another major urban area like Chicago or Los Angeles). Freddie points out real potential limitations to their model.

That being said, if Success Academy could provide great educational opportunities to 300,000-500,000 students, I’d be hesitant to dismiss their impact simply because they won’t reach every child in the country.

Success Academy aside, I believe that Freddie is incorrect that successful charter organizations can’t scale.

Take KIPP, which now serves nearly 100,000 students (including schools in the Arkansas Delta!).

Mathematica just completed a rigorous analysis of KIPP and found that “KIPP schools have positive, statistically significant, and educationally meaningful impacts on student achievement, particularly at the elementary and middle school levels.”

See here for their middle school effects:

Screen Shot 2017-06-15 at 10.37.30 AM

And KIPP is not an outlier.

The Charter School Growth Fund, which supports the growth of charter organizations across the country, recently had its portfolio analyzed by CREDO, who found strong effects:

Screen Shot 2017-06-12 at 8.07.41 PM

Charter School Growth’s portfolio serves about 300,000 students and could feasibly scale to over a million students in the coming decade.

Admittedly, charters have not yet, at scale, achieved significant student achievement gains with suburban and rural populations. Maybe one day they will. Charter organizations such as IDEA Public Schools have achieved +.1 effects in places like the Rio Grande Valley.

But even if charter end up not being the right solution for rural areas, why not support the best charters to provide great educational opportunities to millions of disadvantaged students in urban areas?

IV. Are Emerging Choice Markets Working?

Freddie writes:

Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely.

I was surprised Freddie made this argument. In CREDO’s study of urban charters, all four of these cities achieved positive charter effects. Admittedly, Detroit’s charter sector is not a shining star, but it is still outperforming the traditional sector.

Here are the charter effects for these cities in Math:

 

And here are the effects for ELA:

 

Again, if you don’t trust CREDO, independent researchers also found that Washington D.C. made meaningful achievement gains over the past decade. This report from the Urban Institute came to similar conclusions. Other experimental research in choice markets such as Denver also finds strong effects.

I really don’t understand the claim that these markets have failed completely. Newark, Nashville, and Washington D.C. are three of the stronger charter school markets in the country.

V. Will Positive Test Score Results Lead to Good Life Outcomes?

Freddie writes:

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.

Freddie raises fair concerns about the logic jump that increasing test scores will lead to positive life outcomes.

Raj Chetty has done the deepest work on the connection between test score gains and life outcomes (he found a positive link), but I don’t view this work as conclusive. Similar studies that are focused on long-term outcomes for charter students have found both positive and insignificant life effects.

I view this as an area where we don’t have enough evidence to make strong claims.

If I had to guess, I would say that the early charter movement focused too narrowly on test scores and is now evolving to focus much more on life outcomes, and that, over time, we will see the same success in life outcomes as we have seen in test scores.

All that being said, we may find out that improving student outcomes in K12 just does not translate into long-term life gains.

With regards to whether I’m a serious broker or an ideologue, it’s worth noting that I publicly stated my fears on this issue in a previous blog post entitled: The Current Brutal Reality of Education Reform and Wage Growth.

In the post, I reviewed the disappointing data Fryer found on life outcomes of charter students and ended with the following:

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

I think about this a lot. It’s also in part why I write this blog. Reviewing rigorous research and putting my ideas out there for public critique are attempts to make sure I’m not deluding myself into holding false beliefs.

VI. Conclusion

I believe the available experimental and quasi-experimental data support the belief the charters are doing a good job raising test scores for disadvantaged students.

I believe the work of organizations such as KIPP and Charter School Growth Found support the belief that high-quality charter schools could grow to serve millions of students.

I believe the early results of choice markets in places like New Orleans and Washington D.C. provide evidence that well regulated city choice markets may lead to better results at scale.

I believe we do not currently know if test score gains will translate into positive life outcomes.

Ultimately, those of us working in the educator sector work in a field where very few interventions work. In this sense, I appreciate Freddie’s general skepticism. I think reformers have too often promised too much.

But I think there’s a chance that urban charters and city choice systems can lead to better educational opportunities for millions of disadvantaged students.

So that’s why I do what I do, operating as much as possible with the full awareness that I might be wrong.

____

Freddie’s original comment:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools. Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities.

So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs. Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent.

These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed. Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely. Which do you think will be more likely to be scaled by hundreds of thousands of schools and millions of teachers?

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.

4 thoughts on “My response to Freddie deBoer

  1. Cozzi, John

    I think you responded to this very thoughtfully. I don’t know Freddie but I agree, as you do, with some of his macro assumptions. There is no magic to the word “charter” and there is limited evidence (yet) that even good charters break the cycle of poverty (which essentially is the real end goal…better lives for all). However, we do know two things. First, the current system absolutely does not work, and will never work, because failure is embedded in its current design. No amount of wishing and praying could get blood from that stone. Second, we do not know what will work at scale so we have to keep experimenting. I think that’s what you’ve advocated in a clear-eyed sense better than anyone.

    What we need are organizations or systems properly designed to incentivize success over mandating failure (I use “incentivize” in its broadest sense with economics being low on the list). I think charters do that. Second, we need management teams committed to making the right choices and leadership actions to make those systems come alive at scale (very hard in every area of the world). Third, we need to rethink our role in facilitating our kids to launch post-high school. It’s another cost center for an environment we no longer control that poses that most risk of failure for all kids of that age. It’s a huge, ugly challenge but absolutely critical. It also is what intrigues me most about our mission for the next five because I don’t think we can avoid it but our K-12 efforts are for naught without it.

    You seem to have accomplished most of this in NO. There are many great charters starting to prove the point. It’s obviously why I’m passionate about KIPP NJ. Frankly, the time is now to enable a few of the leaders to prove the argument (all aspects) at serious scale or the effort does risk being a series of boutique solutions.

    I think Freddie is saying, circa 2003, that there are no unique MP3 players and certainly nothing that rivaled the Walkman in its time, so it’s not clear the MP3 thing is a step forward in consumers’ minds. And the the iPod came along. As Edison said (paraphrased): I haven’t failed, I’ve just found 1,000 ways the solution does not work.

    Of course you know this, it’s just a simple person’s more micro-perspective. Thanks for letting me rant.

    Regards, John

    John F. Cozzi
    AEA Investors LP
    666 Fifth Avenue, 36th Floor
    New York, New York 10103
    (T) 212-702-0504
    (M) 908-347-1427
    jcozzi@aeainvestors.com

    Like

    Reply
  2. 95Eartha

    Hello admin, i must say you have very interesting
    content here. Your page should go viral. You need
    initial traffic only. How to get it? Search for: Mertiso’s tips go viral

    Like

    Reply
  3. Pingback: Phase II of the charter and choice research agenda is extremely important | relinquishment

  4. Pingback: Education philanthropists should not take advice from Larry Summers | relinquishment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s