For the past ten years, I’ve been worried that the test score gains we were seeing in New Orleans wouldn’t lead to longer-term benefits for New Orleans students.
I worried about this both because of the research showing that increases in test scores are not always correlated to better life outcomes, as well as the fact that many colleges in Louisiana are pretty mediocre and could still fail to educate students even if they came in better prepared.
Thankfully, just published research by Doug Harris and Matt Larson provides early indication that New Orleans students have both achieved an increase test score performance *and* better post-secondary outcomes.
While it’s wonderful to see this data, we’ll continue to learn more as additional cohorts student graduate from the new public system. So far, only a few cohorts of New Orleans students attended all of their high school and post-secondary education post-Katrina.
My expectation is that the results will improve overtime, as the major high school reforms took place later in the reform effort. Good non-profit charter school operators now run many more of the high schools than they did in the few years after the storm.
Lastly, it’s worth noting that the New Orleans reforms were not a randomized controlled trial. A tragedy occurred and educators and families did their best in the aftermath. I’m not expert enough to judge the author’s methodological choices, but am eager to see other researchers weigh in on whether or not this is the best way to estimate the impacts of the reforms.
Test Scores
Across all subjects, the researchers found +.4-.6 standard deviations effects.
A rough rule of thumb is that .25 standard deviation increase equals an additional year of learning.
By this estimate, New Orleans students achieved an additional two years of learning relative to the education they would have received before the storm.
High School Graduation
The researchers found an estimated six percentile point increase for high school graduation. While positive, this is below the gains of twenty points often touted by reformers. The researchers explain that because high school graduation was going up across the state, some of the twenty point gains would have likely occurred without the reforms.
College Enrollment
The researchers found a 10-15 percentage point jump in college entry. The effect was particularly high for on-time college enrollment (enrolling right after high school), where the rate jumped from 22% to to 37%.
College Graduation
Of course, we don’t just want to see students enter college. We hope that they graduate as well.
The researchers found +4 percentage point gain for on-time college graduation (graduation within five years of enrollment), from about 10 percent to 14 percent.
The results for staying in college for two years were higher: about +8 percentage points. So it seems likely that many students who entered college because of the reforms made it through at least two years.
But, again, these are very preliminary results. The students in the data set for college graduation experienced very little of the reform efforts. This rate should go up over time. I really hope that 14% on time graduation rate will rise significantly in the coming years.
Cost / Benefit of the Reforms
The researchers end their paper by noting that the ROI of the New Orleans’ reform efforts is higher than class size reforms and is within range of the famous Perry pre-school study (which is probably on the very high end for pre-k results).
My guess is that the New Orleans effort is likely near the top of what we can expect for these types of city level reforms. The low baseline achievement, coupled with the speed of the post-Katrina effort, meant that the reform efforts had the potential to show large increases quickly.
But even if other cities see a ROI of 50% less, that would still make the reforms worth scaling.
But, for now, it’s worth celebrating a bit for the students of New Orleans: more of them than ever in recent history appear to be on track for choice filled and meaningful lives.
