Category Archives: Achievement School District

Doubling down when you’ve been wrong

From 2012 to 2016, three reforms took place in Memphis: the Gates Foundation teacher evaluation reform, the Achievement School District (ASD) charter school turnaround effort, and the district run izone turnaround effort.

In private conversations, I predicted that neither the teacher evaluation nor izone efforts would work. I thought the ASD charter effort work would work. The Arnold Foundation funded research to help us understand if our beliefs about how to help children were correct.

Five years later, the results paint a different picture than my predictions: the teacher evaluation work did not improve achievement. The ASD charters have yet to deliver results. And the izone schools have done the best.

I was wrong about the izone and the ASD.

So what to do now?

The short answer: expand non-profit governance to protect the gains of izone schools, and continue to grow the best non-profit public charters and replace the worst.

The longer answer is below.

Responding to Being Wrong

There are three ways to respond to being wrong: ignore that you were wrong, change your behavior, or admit that you were wrong but double down.

Ignoring that you’re wrong is never good.

Changing your behavior is often the most reasonable response.

Doubling down comes with substantive and reputation risk, but it can be the right thing to do.

Memphis Results 

The teacher evaluation reforms went roughly as I expected, so I won’t discuss that too much here, other than saying that my predictions were based on the belief that execution would hamper implementation in big urban districts, which seems to be exactly what happened.

Here are the results of the izone and ASD charter efforts:

 

 

This slideshow requires JavaScript.

The izone had a positive .2 stand deviation effect over five years. This is fairly good impact and it’s great to see these schools helping students.

The ASD schools had no impact.

What I Got Wrong

I made a few mistakes.

First, I underestimated the school district’s strategic and executional competency. In creating the izone, the district successfully recruited a lot of their best educators to their izone schools, and then appointed a talented leader, Sharon Griffith, to oversee these schools. This strategy (concentrating talent) and execution (selecting a great leader) worked in ways I did not expect.

Second, I overestimated the abilities of charter networks to deliver results in the first few years of the turnaround efforts. My assumption was that expert charter leaders would be able to deliver better results pretty quickly, even though they were going into very difficult situations and were often coming from out of town. Unfortunately, a few operators really struggled in the early years, with one of them, Yes College Prep, choosing to not even open up a school (this was a big hit as YES is considered to be one of the best networks in the country).

What I Would Have Done Differently 

I should have realized that great educators in the traditional system could achieve good results if given autonomy in a competitive environment. I have changed my beliefs on this issue based of the work of the izone, the work in Lawrence, Massachusetts, and my lived experience of seeing great teachers and leaders in all types of schools.

Rather than predict the izone wouldn’t work, I should have predicted that the izone might work, but advocated for non-profit governance over the zone so that it could sustain its gains. This remains my worry with the izone: that it will not be able to sustain its impacts over the next 5-10 years due to its governance structure, which makes it vulnerable to political shifts.

I also worry that it can’t really scale. Putting the best educators in the worst school is more of a one time intervention rather than a strategy that can grow over time, and it has some negative effects on the schools that lose their great educators.

Second, I should have realized that early stage charter operators (either newly started or new to Memphis) would struggle to scale through full school turnarounds in a difficult political environment. In hindsight, I should have advocated for more one grade at a time roll outs of newer operators. While this would have been more disruptive to families at the outset, it would have been better for kids over the long run.

What to Do Now

If the above diagnosis is true, it doesn’t really call for a dramatic change in strategy.

Rather, it calls for trying to get non-profit governance over the best district schools, growing the charters that are working, replacing those that aren’t, and starting new non-profit district and charter schools under more favorable conditions.

Getting non-profit governance to the best district educators would (I think) require a change in legislation.

Growing the best charter schools and closing the worst  can be done under current policy.

Evidence to Support Doubling Down

Researchers have found evidence of charter school sectors improving over time. This paper on charters in Texas found significant sector improvements, which led the researchers to write:

The findings suggest the value of taking a longer-term perspective when evaluating the impact of a major educational reform such as the introduction of charter schools, especially when the success of the reform ostensibly depends on parental decisions and market forces.

Additionally, previous research on Memphis charters found strong positive effects:

Screen Shot 2018-06-23 at 8.03.44 AM

These results mirror the results of many other urban charter sectors, which generally find positive effects.

So there is a research based case for predicting that the Memphis charters will improve over the next 5-10 years. We have already seen this with some operators, such as Aspire.

In Sum 

The research has increased my belief that great district school educators, with autonomy and support (and often pressure on local districts from the state to give this autonomy) can really help struggling students.

I still believe that these gains risk backsliding due to political shifts, so I support giving these schools non-profit governance so they can build enduring organizations (both Denver and Indianapolis have mechanisms to give district schools non-profit status).

I also still believe that growing the best charters and closing the worst is one of the best hopes for improving student achievement in Memphis.

So, in many ways, I’m choosing to double down despite being wrong, with the caveat of calling for non-profit governance for more types of public schools.

While this is uncomfortable, I don’t know a better way forward. I think doubling down will help children.

I understand that many of those who I disagree with say “we just need more time.” And I’m making a version of this argument here. So I’ve tried to lay out all my assumptions as clearly as possible, both for the sake of transparency, as well as with the hope that these assumptions can be corrected if they are wrong.

Brown University vs. Science

The Annenberg Institute for School Reform at Brown University just released a report on state turnaround districts.

Brown University 

The report states:

In short, the Recovery School District, which was marketed (and continues to be lauded) as ushering in a miraculous transformation in New Orleans, has not kept its promise to some of the country’s most disadvantaged students.

The report cites another report, from Stanford University, and makes the following claims about equity and accountability:

The SCOPE [Stanford center] review also found that school quality and accountability are impeded by the lack of a strong central system (within the RSD) to support instructional improvement or maintain safeguards to ensure equity and access to reasonable quality of education.

Science 

Here is what Doug Harris, who actually studied student achievement in New Orleans, wrote:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Here’s a graph that captures these gains:

Screen Shot 2015-08-04 at 7.03.28 AM

Equity 

As for equity, I think this has been NOLA’s greatest innovation. I wrote a report on it. Here’s a highlight: despite serving a very at-risk student population, New Orleans has a lower expulsion rate than the state.

Screen Shot 2015-12-21 at 3.46.37 PM

Accountability 

As for accountability, it’s hard to think of a city that has been more serious about ensuring students don’t attend failing schools. In 2004, 60% of New Orleans students attended a school that was in the bottom 10% of the state. Now 13% do.

Screen Shot 2015-12-21 at 3.53.44 PM

People Who Live in New Orleans 

Lastly, here’s what New Orleanians think about the reforms:

Reasonable people can debate whether or not other states will see the same results.

But to say that the RSD has not kept its promise to the country’s most disadvantaged students is not supported by science.

Of course, there is still an incredibly long way to go in New Orleans. ACT scores, for example, are at an all-time high at around 19, but this still falls short of college and career ready.

Also, the New Orleans reforms were messy. While the academic results are undeniable, it’s been ten years of tense and difficult work, with many mistakes made along the way.

But if you don’t want things to be messy, you’re in the wrong line of work. The issues or race, class, and poverty are insanely complicated. If you work in the sector and haven’t changed your mind about a major issue, then you’re probably not thinking deeply enough.

All that being said… the student achievement gains are real. Children are better off.

And students across the country would be much better off if other cities achieved results similar to those in New Orleans.

Hopefully this will occur.

The Not So Scary Lessons of Recovery and Achievement School Districts

Around ten years ago, the Recovery School District (RSD) assumed governance over all of the failing schools in New Orleans.

Around four years ago, the Achievement School District (ASD) assumed governance over many of the failing schools in Memphis.

Recovery School District 

Here is the official presentation that the RSD just made to the state school board. Note that the data is presented as a RSD pre / post comparison, not city as a whole. Some highlights:

First, lest you think it’s because student body changed dramatically:

Screen Shot 2015-07-29 at 7.52.18 PM

Proficiency:

Screen Shot 2015-07-29 at 7.53.01 PM

ACT:

Screen Shot 2015-07-29 at 7.53.43 PM

State based college scholarship eligibility:

Screen Shot 2015-07-29 at 7.54.49 PM

Lastly: none of the above qualify as rigorous research results (though I still think they are very useful, illustrative, and paint an accurate picture). But, as readers of this blog will know, next month Doug Harris is going to release a study that shows that NOLA reforms achieved .2-.4 effects at an ROI that surpasses that of pre-k and class size reduction.

The RSD worked.

Much still to be done, of course.

Achievement School District

See here for a deck on the ASD’s results.

Unlike Louisiana, Tennessee actually has a statistically rigorous state based measurement: TVAAS (a growth measure).

School can achieve a TVAAS rating of 1-5, with 5 being the highest.

If all ASD schools in their second and third years were considered districts, they would each receive a 5, the highest TVAAS score.

Screen Shot 2015-07-29 at 8.00.44 PM

Note that first year schools scored a 1 on TVAAS.

The ASD has a hashtag for this phenomenon: #MoreTimeMoreGrowth.

Read the whole deck for the rest of the data. There a lot of bright spots, as well as a few worrisome trends (reading scores).

Looking Ahead 

The RSD worked, though under unique circumstances.

The ASD is on the right track, though it’s still early.

The Education Achievement Authority (EAA) of Michigan, continues to struggle.

Why?

Because the RSD and ASD are building great ecosystems of high-peforming operators (though, to it credit the ASD direct schools are doing great, a testament to Chris Barbic’s ability to get 10,000 things done at once). But the long-run ASD game is still in non-profit operator management.

The EAA attempted to direct run all it schools.

Directing running all your schools runs afoul of this data on the performance of urban charter schools.

Screen Shot 2015-07-27 at 4.31.22 PM

Currently, Nevada and Georgia are in the midst of enacting RSD / ASD models.

I hope they will learn from the above lessons. In Georgia, this will be easier to do, as there is already a base of decent charter operators. Nevada, unfortunately, has one of the worst performing charter sectors in the country. Perhaps they can use this opportunity to grow a higher-quality supply as well close some of their low performers.

Don’t Be Scared

Over the past week, I’ve been harping (I hope not trolling!) on Libby Nelson’s piece on the scary lesson of No Child Left Behind:

Screen Shot 2015-07-29 at 8.14.29 PM

As I mentioned before, I think Libby is a good reporter.

The only reason I’m hammering this point is because I believe it’s so important to the future education of millions of low-income children.

We do have very good ideas about how to fix failing schools. And when reporters cover this issue, they should point to the effectiveness and scale of urban charter schools.

Urban charter schools are working. Governance interventions such as the RSD and ASD have enabled high-performing charters to grow.

Scaling these reforms will continue to be a difficult path.

But it’s not a scary path.

It’s a hopeful one.