Category Archives: New Orleans

Early signs that the New Orleans reforms did more than raise test scores

For the past ten years, I’ve been worried that the test score gains we were seeing in New Orleans wouldn’t lead to longer-term benefits for New Orleans students.

I worried about this both because of the research showing that increases in test scores are not always correlated to better life outcomes, as well as the fact that many colleges in Louisiana are pretty mediocre and could still fail to educate students even if they came in better prepared.

Thankfully, just published research by Doug Harris and Matt Larson provides early indication that New Orleans students have both achieved an increase test score performance *and* better post-secondary outcomes.

While it’s wonderful to see this data, we’ll continue to learn more as additional cohorts student graduate from the new public system. So far, only a few cohorts of New Orleans students attended all of their high school and post-secondary education post-Katrina.

My expectation is that the results will improve overtime, as the major high school reforms took place later in the reform effort. Good non-profit charter school operators now run many more of the high schools than they did in the few years after the storm.

Lastly, it’s worth noting that the New Orleans reforms were not a randomized controlled trial. A tragedy occurred and educators and families did their best in the aftermath. I’m not expert enough to judge the author’s methodological choices, but am eager to see other researchers weigh in on whether or not this is the best way to estimate the impacts of the reforms.

Test Scores

Across all subjects, the researchers found +.4-.6 standard deviations effects.

A rough rule of thumb is that .25 standard deviation increase equals an additional year of learning.

By this estimate, New Orleans students achieved an additional two years of learning relative to the education they would have received before the storm.

Screen Shot 2018-07-22 at 8.30.07 AM

High School Graduation

The researchers found an estimated six percentile point increase for high school graduation. While positive, this is below the gains of twenty points often touted by reformers. The researchers explain that because high school graduation was going up across the state, some of the twenty point gains would have likely occurred without the reforms.

Screen Shot 2018-07-22 at 8.32.47 AM.png

College Enrollment

The researchers found a 10-15 percentage point jump in college entry. The effect was particularly high for on-time college enrollment (enrolling right after high school), where the rate jumped from 22% to to 37%.

Screen Shot 2018-07-22 at 8.39.13 AM.png

College Graduation 

Of course, we don’t just want to see students enter college. We hope that they graduate as well.

The researchers found +4 percentage point gain for on-time college graduation (graduation within five years of enrollment), from about 10 percent to 14 percent.

Screen Shot 2018-07-22 at 8.40.27 AM

The results for staying in college for two years were higher: about +8 percentage points. So it seems likely that many students who entered college because of the reforms made it through at least two years.

But, again, these are very preliminary results. The students in the data set for college graduation experienced very little of the reform efforts. This rate should go up over time. I really hope that 14% on time graduation rate will rise significantly in the coming years.

Cost / Benefit of the Reforms

The researchers end their paper by noting that the ROI of the New Orleans’ reform efforts is higher than class size reforms and is within range of the famous Perry pre-school study (which is probably on the very high end for pre-k results).

Screen Shot 2018-07-22 at 8.55.14 AM

My guess is that the New Orleans effort is likely near the top of what we can expect for these types of city level reforms. The low baseline achievement, coupled with the speed of the post-Katrina effort, meant that the reform efforts had the potential to show large increases quickly.

But even if other cities see a ROI of 50% less, that would still make the reforms worth scaling.

But, for now, it’s worth celebrating a bit for the students of New Orleans: more of them than ever in recent history appear to be on track for choice filled and meaningful lives.

The New Orleans reforms have been both impactful and popular but the gains are plateauing

The New Orleans reform efforts are nearly thirteen years old.

As a reminder, the efforts led to some of the most significant achievement gains in our country’s recent history.

ednext_XV_4_harris_fig01-small

In a recent Sean Reardon study that appeared in the New York Times, New Orleans was the only city that scored in the top ten for growth in the country and serves a majority of African-American students.

All of the other top performing districts, except for Chicago, barely serve any African-American students, as the chart of the top ten growth districts below shows:

nola AA.png

College enrollment rates have also skyrocketed, though we don’t yet have great data on college completion and post-secondary outcomes.

college enrollment NOLA

However, most of the test score gains were made in the first 7 years of the reforms, and the city is no longer increasing relative to overall state performance.

Screen Shot 2018-06-07 at 8.57.20 AM

The gains in New Orleans are very real. But so is the current plateauing of test score performance. It remains an open question whether or not the system will see another significant increase in student performance. My guess is that the increases will eventually pick up again, but at more modest rates.

A particularly difficult strategy question is what philanthropy can do to help New Orleans at this juncture.

Leaders in the city are both trying to double down on expanding the best operators, as well as help all schools increase their instructional rigor.

Over the long-haul, I believe that most gains will come from scaling the best school operators, selectively starting new operators, and replacing the worst. But I do think that supports to help all schools can lead to some improvement, though my expectations with these types of reforms are modest. Many smart people disagree on how to allocate funds across these two types of strategies. It will be interesting to see what we learn from New Orleans over the next few years on this issue.

____

But what do the people think?

The Cowen Institute just came out with its annual polling data on New Orleans education. The poll draws from registered voters in New Orleans.

When it comes to voter perception of the system, votes are split between “the system is getting better” and “the system is staying the same.” Very few feel it’s getting worse.

Screen Shot 2018-06-07 at 9.03.53 AM

The perception mirrors the data: the system has improved, but these improvements are slowing down, and it’s unclear that things are getting much better now.

The public also continues to support charter schools and the city’s unified enrollment system.

Screen Shot 2018-06-07 at 9.06.49 AM

Only 17% of voters disagreed that charters had improved education, putting charter favorability at a 3:1 ratio.

Even more interesting: 70% of public school parents believed charter schools had improved education, compared to only 50% of those without children.

Charters seem to be winning those they serve but not fully winning those they don’t.

Open enrollment also only had 16% negative rate, putting it a 3:1 favorability rating.

___

Overall, the New Orleans reforms have been both impactful and popular.

Right after Katrina, neither of these outcomes were inevitable. In the early years of the reforms, school performance was very uneven and the reforms were very controversial.

It is incredibly difficult to transform a whole city’s educational system in a way that increases opportunities for children and garners the support of the public.

I am hopeful that other cities, such as Camden (see here for a good New York Times profile on the city’s efforts), will also do great things for kids and gain the public’s trust.

 

 

Most students in NOLA whose school is closed end up in one of their top choices for the following year. Here’s how.

This post from Ed Navigator is worth reading. It covers schools closures in New Orleans.

Over ten years after Katrina, and under an elected school board, New Orleans continues to selectively close underperforming schools.

I view this as a good thing, given the growing body of research that shows that school closures help kids when the students end up in better schools.

New Orleans uses a unified enrollment system to help kids get into better schools.

The unified enrollment system gives preference to students whose schools were closed the year before. If your school was closed, the algorithm bumps you to the top of the list for any school you want to get into.

Ed Navigator works with families whose schools have been closed, so that they can help select great schools.

The result?

This year, 87% of students who attended a closing school and used the enrollment system received on of their top three choices for the next school year.

94% of the students will now attend a school that is rated higher by the state’s grading system.

The system is by no means perfect. My biggest critique is that the state’s grading system still relies too heavily on absolute test scores (rather than growth). I also understand the counterarguments that government should never close schools and should instead let enrollment patterns (driven by parental choice) determine which schools grow and which close.

But I would rather have the New Orleans enrollment and closure system than just about any other big city system in the country.  In too many cities, really bad schools stay open for too long. And if anything happens to them, kids often end up in schools that are just as bad.

This is not what happens in New Orleans.

It’s also great to see Parag Pathak (and his colleagues) work in action. Parag recently won the John Bates Clark award in part because of his contributions to working on unified enrollment systems.

It’s rare that an idea goes from the ivory tower to think tanks to actual implementation by a democratically elected body to  helping citizens.

This is really great to see. And really great for kids in New Orleans.

The Atlantic vs. Facts

I’m sympathetic to how hard it is for journalists to cover complicated policy areas.

Translating policy into readable articles that both inform and entertain is difficult. In this sense, writing a policy a blog is a luxury. For the most part, I just get to stick to the research and ideas.

But journalists do have a duty to accurately cover the basic facts of the issue they are covering.

In her recent article on Japan’s education system, Alana Semuels included a throw away line about New Orleans education reforms; she writes:

The equity in Iitate stands in stark contrast to a place like New Orleans, which was also hit by a disaster. While Japan’s national government tried to ensure that students in the affected area got more resources after the accident, officials in New Orleans disinvested in the public educational system in their city. Public-school teachers were put on leave and dismissed, many students disappeared from schools’ rolls, and the New Orleans system now consists almost entirely of charter schools.

If you click on the links, which presumably justify her claims, you don’t get taken to supporting research; instead, you get taken to an op-ed.

As it happens, rigorous research has been done on the New Orleans reforms.

In terms of student achievement, after Katrina New Orleans saw some of the biggest gains ever recorded in an urban school district. You can read about the research here.

In the article, the authors of the study note:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Rigorous research has also been done on how public spending in education changed in New Orleans after Katrina. Contrary to Semuels claim, spending went up.

The authors of the study write:

New Orleans’ publicly funded schools spent 13% ($1,358 per student) more per pupil on operating expenditures than the comparison group after the reforms, even though the comparison group had nearly identical spending before the reforms.

A quick google search could have turned up both of these studies.

I wish Semuels had taken the time to review these studies.

Because she did not, thousands of readers have been misinformed.

Applying portfolio reforms to postsecondary problems

I just got back from a trip to New Orleans, which continues to be a well of friendship and inspiration.

I. Where should you spend the next philanthropic dollar? 

In a few conversations, the following questions came up:

  1. Are the kids we serve going to succeed in life after high school? What will their lives be like when they are 30? Will they be living meaningful and happy lives?
  2. Is the marginal dollar of philanthropy best spent on making the K-12 system better (after 10 years of improvements) or trying to overhaul the post-secondary landscape?
  3. If you wanted to radically improve post-secondary, what would you do?

II. Post-Secondy portfolio 

The K-12 portfolio mindset entails viewing an educational system in terms of operators (running schools) and seats (how many students are served).

This mindset could also be applied to post-secondary.

By 2020 or so, New Orleans will be graduating around 3,000 students a year.

Let’s say that about 1,500 of them will be prepared to succeed in a four year college; 1,000 of them will be prepared to succeed in a 1 to 2 year credentialing program; and 500 of them will need deep support to enter the workforce and exit crisis situations.

Of the four year college students, you might need 500 to 1,000 “KIPP to College” type supports to ensure students make it through.

For the credentialing programs, you’d need 1,000 seats that can reliably produce students with employable credentials.

For the crisis students, you’d need employment and social service operators that could transition students into jobs.

III. Post-Secondary investment intermediaries 

Instead of assuming this will naturally happen in New Orleans (or any other city), you could capitalize a new or existing non-profit intermediary to launch, recruit, and support post-secondary providers.

At the outset, the intermediary would create a business plan where it laid out how money it would need to get X% coverage on the aforementioned 3,000 seats.

High-Quality existing local providers (like the coding bootcamp Operation Spark) could cover some of the seats, and national providers like Match Beyond could be recruited in.

Overtime, you’d expand what was working, close what wasn’t, and support new entrepreneurs to keep innovation going.

IV. Getting funding streams right

Most states subsidize mediocre public universities; the federal government tops this off with Pell grants.

To make the 3,000 seat post-secondary strategy viable, you’d need to blend a mixture of public support and tuition to make providers sustainable.

Louisiana’s course choice provides a revenue stream for programs that started working with kids while they’re in high school.

Creating a new university that housed many of these programs could allow for the accessing of Pell grants.

Wage contigent loan programs could also be an option for programs that were consistently placing graduates in high-performing jobs.

V. Who are the entrepreneurs that will seek out the 10x play?

The early New Orleans K12 entrepreneurs felt that they could deliver something to students that was significantly better than the existing system.

They were right.

A post-secondary transformation won’t happen on its own.

It will take a set of entrepreneurs to put forth a plan, galvanize funding, and spend a decade building the new system.

Is this the right play? If so, who will step up?

What if everything you believe about education is wrong?

When I first started think about increasing educational opportunity, I had a lot of beliefs, and, given the opportunity, I was willing to act on any of them.

Now I have fewer beliefs and am only willing to act on them under very specific circumstances.

I. Acting on Limited Beliefs in New Orleans  

As an undergrad and law student, I had a lot of beliefs on education policy. If you named an education policy topic, I had a strong opinion.

By the time I was working deeply in New Orleans, I was, for the most part, only acting on two of these beliefs:

(1) Starting and scaling high-quality charter schools (generally based on the high expectations / high support model); and

(2) Recruiting as much high-quality talent as possible (generally measured by leadership potential, undergraduate selectivity, and GPA).

Those seemed like the two most important near-term problems to solve in New Orleans.

Eventually, I also started working on systemwide regulatory issues, as we had to figure out how to govern an all charter school system.

II. So Much Conventional Wisdom (on All Sides) Was Wrong 

10 years later, we’ve a learned that New Orleans achieved dramatic student achievement results while, at the same time, upending much of the conventional wisdom of education policy.

The Education Research Alliance has done a bunch of great research that should humble pundits and educators alike; some highlights below:

Popular Belief #1: Experienced Teachers are Better

New Orleans increased student achievement while the % of teachers with less than 5 years of experience skyrocketed, and the % of teachers with over 20 years of experience plummeted.

screen-shot-2017-01-17-at-10-35-39-am

Popular Belief #2: Teacher Credentialing Matters

New Orleans increased student achievement while the % of teachers with no, temporary or, the lowest level of certification (C/1) skyrocketed, and the % of teachers with advanced credentials (A/3 and B/2) plummeted.

screen-shot-2017-01-17-at-10-39-22-am

Popular Belief #3: Teacher Turnover is Bad

New Orleans increased student achievement while teacher turnover significantly increased.

screen-shot-2017-01-17-at-10-53-14-am

Popular Belief #4: Education Choice Markets Will Fail Due to Information Problems

A continuing criticism of choice based reforms is that parents will make poor decisions due to being incapable of understanding school quality. In New Orleans, student achievement (SPS score) was one of only a few factors that strongly increased the likelihood a family would choose a school in the open enrollment system.

screen-shot-2017-01-17-at-10-56-29-am

Popular Belief #5: School Choice Only Benefits a Select Group of Choosers

A continuing criticism of choice based reforms is that only the active choosers will get into the schools they want. In New Orleans, 75% of families were matched with one of their top three choices.

screen-shot-2017-01-17-at-11-07-16-am

Popular Belief #6: School Choice Increases Student Mobility

Numerous commentators have argued that school choice will cause significant increases in student mobility. New Orleans moved to an all choice system and student mobility decreased.

screen-shot-2017-01-17-at-11-12-31-am

Popular Belief #7: School Closure Harms the Children Attending the Closed School

In New Orleans, students attending schools that were closed (intervention schools) saw their student achievement increase significantly in the subsequent years.

screen-shot-2017-01-17-at-11-16-27-am

Popular Belief #8: Money is Best Spent “in the Classroom”

In comparison to other districts, New Orleans increased student achievement while spending more on administrative costs and less on instructional costs.

screen-shot-2017-01-17-at-11-20-29-amscreen-shot-2017-01-17-at-11-20-36-am

III. We Might be Wrong About More

To be clear, I’m not saying that any of the above caused the student achievement gains in New Orleans.

Rather, I’m only pointing out that New Orleans saw some of the most dramatic student achievement gains in our country’s recent history while doing a bunch of things that you’re not supposed to do.

I’m sure we’ll learn more over the coming decade.

Lastly, New Orleans is one city with a very unique set of circumstances, we don’t yet know how similar types of reforms will work in other cities.

IV. Keep Your Policy Identity Small and Your Problem Solving Identity Large 

The more strongly held beliefs you have on education policy, the more likely you are to (1) be wrong (2) find it hard to admit you’re wrong and (3) mandate broad solutions that don’t work in many circumstances.

The more strongly oriented you are toward solving problems, the more likely you are to (1) be right in your specific circumstance (2) change your mind if you’re wrong and (3) be willing to admit that your solution might not work in other contexts.

V. Help Build Systems that Increase Problem Solving Capacity

Given how much we don’t know, as well how many problems are context specific, one of our primary policy goals should be to experiment with system level structures that increase the power and ability for educators and families to solve problems.

I view New Orleans as one example of how a system can be structured to empower families and educators to solve problems.

Hopefully we’ll build more examples over the coming years.

NOLA Return Bill: Is it Good?

Screen Shot 2016-05-13 at 2.09.37 PM

So far, many articles focused on the new bill have concerned themselves with raising or lowering the status of specific players – but what about the bill itself?

___

Overall, I think the bill is strong on equity, democracy, and protecting existing academic gains.

I worry about what the future holds for maintaining a sense of urgency on increased academic performance and innovation. A cementing of structure will likely favor the incumbents.

But a law can only do so much.

A good framework has been set. The return bill enshrines much of what has led to NOLA’s gains in equity and performance.

In a world of toxic national politics, it’s heartening to see complex and important legislation being crafted and adopted by a politically and racially diverse coalition.

___

More reflections:

  1. This is nationally important: we now have a law on the books that clearly defines how a portfolio district can be implemented. A debt is owed to Paul Hill.
  2. The bill is strong on equity: the weighted student funding, enrollment, and expulsion provisions are extremely important. As I’ve written before, I think NOLA’s greatest innovations have been in equity.
  3. The bill is sticky: with fairly broad support, the bill codifies 10 year’s worth of work – this should make it stick. Moreover, the marketshare limits and charter autonomy provisions should prevent roll backs to a one best system.
  4. The return is really a rebalancing: the school board now has much more power, and the state has much less – however, the state still has the RSD as a check on the local school inaction on underperforming schools. Democratic power exists at both local and state levels, and this is a readjusting rather than a jettisoning of state power.
  5. Who will be the leaders? Many of the local leaders in education reform have been unelected: non-profits and civic organizations have held a lot of power. This will shift as the local school board gains more power, but with elections happening this fall, we don’t yet know who this slate of leaders will be. The first wave of local leaders post-return will set a vision and culture for what local control means…
  6. Incumbents are protected – but what about future schools? The bill outlines clear powers for the school board, superintendent, and existing schools, but less ink was spilled on ensuring there is a continually pipeline of new (and hopefully innovative) schools. I think this amongst the biggest risks in NOLA over next decade: will the incumbents of the system (government, charter schools, non-profits, etc). utilize their hard and soft power to block new entrants?

___

Lastly, here are what I think to be the most important parts of the bill – with excerpts of the exact language for those who want to understand the mechanics.

Equity Provisions 

Student based budgeting w/ weights for specific needs: “…establishes a process to determine the district-level funding allocation to be effective beginning July 1, 2017, and as revised in subsequent years as appropriate, based upon student characteristics or needs…”

Unified enrollment and expulsion: “…shall require all charter schools under the board’s jurisdiction to participate in the parish-wide enrollment system and student expulsion process, according to policies established by the board…”

Allows for some (but not exclusive) neighborhood preference: “May provide a lottery preference for enrollment at elementary middle schools under the board’s jurisdiction for students residing with defined geographic zones as one of the factors to determine student assignment, according to policies adopted by the board. Such preference shall be applied to not more than one-half of the seats available in each grade level…”

Promotes integration: “…so that such schools shall be exempted from the minimum enrollment percentages…”

Performance Provisions

Prevents monopoly / too big too fail: “shall adopt a policy establishing a process which allows the local superintendent to limit the percentage of system enrollment that any single operator of schools or charter governing authority may serve to ensure that a diverse system of schools led by multiple high quality operators exists at all times.”

Empowers superintendent as portfolio manager: “superintendent shall present recommendations to the local school board regarding the approval, extension, renewal, or revocation of the charter for any charter school under the board’s jurisdiction…Unless vetoed by a two-thirds vote of the full membership of the board, the local superintendent may implement any such recommendation submitted to the board.”

Protects charter autonomy: “the local school board shall not impede the operational autonomy of a charter school under its jurisdiction.”

Promotes test security: “each charter school under the local school board’s jurisdiction shall provide for independent test monitoring from third-party entity approved by the school board for the testing period immediately preceding the board’s consideration of renewal of the charter school’s contract.”