Early signs that the New Orleans reforms did more than raise test scores

For the past ten years, I’ve been worried that the test score gains we were seeing in New Orleans wouldn’t lead to longer-term benefits for New Orleans students.

I worried about this both because of the research showing that increases in test scores are not always correlated to better life outcomes, as well as the fact that many colleges in Louisiana are pretty mediocre and could still fail to educate students even if they came in better prepared.

Thankfully, just published research by Doug Harris and Matt Larson provides early indication that New Orleans students have both achieved an increase test score performance *and* better post-secondary outcomes.

While it’s wonderful to see this data, we’ll continue to learn more as additional cohorts student graduate from the new public system. So far, only a few cohorts of New Orleans students attended all of their high school and post-secondary education post-Katrina.

My expectation is that the results will improve overtime, as the major high school reforms took place later in the reform effort. Good non-profit charter school operators now run many more of the high schools than they did in the few years after the storm.

Lastly, it’s worth noting that the New Orleans reforms were not a randomized controlled trial. A tragedy occurred and educators and families did their best in the aftermath. I’m not expert enough to judge the author’s methodological choices, but am eager to see other researchers weigh in on whether or not this is the best way to estimate the impacts of the reforms.

Test Scores

Across all subjects, the researchers found +.4-.6 standard deviations effects.

A rough rule of thumb is that .25 standard deviation increase equals an additional year of learning.

By this estimate, New Orleans students achieved an additional two years of learning relative to the education they would have received before the storm.

Screen Shot 2018-07-22 at 8.30.07 AM

High School Graduation

The researchers found an estimated six percentile point increase for high school graduation. While positive, this is below the gains of twenty points often touted by reformers. The researchers explain that because high school graduation was going up across the state, some of the twenty point gains would have likely occurred without the reforms.

Screen Shot 2018-07-22 at 8.32.47 AM.png

College Enrollment

The researchers found a 10-15 percentage point jump in college entry. The effect was particularly high for on-time college enrollment (enrolling right after high school), where the rate jumped from 22% to to 37%.

Screen Shot 2018-07-22 at 8.39.13 AM.png

College Graduation 

Of course, we don’t just want to see students enter college. We hope that they graduate as well.

The researchers found +4 percentage point gain for on-time college graduation (graduation within five years of enrollment), from about 10 percent to 14 percent.

Screen Shot 2018-07-22 at 8.40.27 AM

The results for staying in college for two years were higher: about +8 percentage points. So it seems likely that many students who entered college because of the reforms made it through at least two years.

But, again, these are very preliminary results. The students in the data set for college graduation experienced very little of the reform efforts. This rate should go up over time. I really hope that 14% on time graduation rate will rise significantly in the coming years.

Cost / Benefit of the Reforms

The researchers end their paper by noting that the ROI of the New Orleans’ reform efforts is higher than class size reforms and is within range of the famous Perry pre-school study (which is probably on the very high end for pre-k results).

Screen Shot 2018-07-22 at 8.55.14 AM

My guess is that the New Orleans effort is likely near the top of what we can expect for these types of city level reforms. The low baseline achievement, coupled with the speed of the post-Katrina effort, meant that the reform efforts had the potential to show large increases quickly.

But even if other cities see a ROI of 50% less, that would still make the reforms worth scaling.

But, for now, it’s worth celebrating a bit for the students of New Orleans: more of them than ever in recent history appear to be on track for choice filled and meaningful lives.

2 thoughts on “Early signs that the New Orleans reforms did more than raise test scores

  1. kjbonill

    Did Joseph Goebbels and Alan Turing have a child? The use of unrelated data sets pieced together to justify the monetization of New Orleans public schools is haute-propaganda. Absent a measurable and meaningful barometer for success, the author plays a shell game to declare corporate education the winner. Is speed of college graduation, half-completion, or even high school graduation evidence of success? Do profit-driven companies have vested financial interest in manufacturing positive numbers? Hmmm.
    That same mistake was made in Vietnam. Whether our foreign policy in Southeast Asia, much like the education of children, is successful lies beyond the capacity of any algorithm. Because we couldn’t (and can’t) measure what matters, we make what we can measure matter. In Vietnam, that meant that body count measured success, despite the reality of creating more enemies and destroying the country. In education, our fetishization of data means we measure equally irrelevant data with the same counterproductive results. Standardized test scores, like body counts, fail to measure educational successes. Further, they fuel policies antithetical to real learning. Education should mean much more than growth measurable by data. Education should strive to empower human beings to achieve self-actualization. There will never be profit in that.

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s