Baton Rouge’s charter school turnaround

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter and innovation schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

The results just came in for Baton Rouge.

The Historical Underperformance of Baton Rouge’s Charter Sector 

After Hurricane Katrina, New Orleans’ public charter schools drove impressive achievement gains for the city.

Baton Rouge civic leaders and educators tried to replicate this strategy. It didn’t work. The charter schools that opened performed poorly.

A typical response to these weak results would have been to abandon the strategy.

But this is not what happened in Baton Rouge.

Civic leaders, led in large part by the leadership of New Schools for Baton Rouge, came to the conclusion that the charter strategy was right but that the execution was flawed.

So they began the hard work of recruiting new charter leaders as well as working with struggling schools to wind down their operations.

7 Years Later

New Schools for Baton Rouge was founded in 2012. It took many years to raise money, build a civic coalition, and recruit great educators.

Now, in 2019, we’re beginning to see the results of the work.

The slides below are a bit hard to read, but they capture the achievement effects of magnet, charter, and traditional schools in Baton Rouge between 2014 and 2017.

The study compared students in these sectors to their peers across the state (controlled for demographics and other salient factors).

Math Effects

Screen Shot 2019-05-11 at 9.08.07 AM

Reading Effects

Screen Shot 2019-05-11 at 9.07.56 AM

If you look at the small dotted lines tracking the [C] results, you’ll see major improvements in the Baton Rouge charter sector.

In 2014, the charter sector had -.13 effects in math (yellow lines) and -.16 effects in reading (blue lines).

These are pretty bad results.

But only a few year later the charter sector had gone from negative effects to neutral effects.

Improving an entire sector by ~.15 standard deviations in just three years is an impressive feat. Keep in mind that New Orleans achieved ~.4 standard deviation impact in the six years after the storm; so to grow .15 standard deviations in just three years is nearly on par with the growth we saw across the city in New Orleans.

While test scores aren’t everything, it’s rare that schools with very negative test effects deliver positive long-term life outcomes.

So this turnaround of the Baton Rouge charter sector is wonderful to see.

Lessons for Other Cities

In CREDO’s study of urban charter sectors across the country, around 20% of cities had negative effects for their charter sectors.

If I were a leader of one of these cities, I’d get on a plane and take a trip to Baton Rouge.

Baton Rouge leaders are showing that it’s possible to turnaround a city’s charter sector. The strategy is fairly simple. Open up annual cohorts of good schools and replace underperforming schools.

But the execution is difficult.

It took a great board and team at New Schools for Baton Rouge to give good charter operators the confidence they could grow in a city that had a struggling sector.

It took major investments in teachers and leaders to grow the talent base to support operator growth.

And it took a coalition of advocacy organizations and political leaders to create the policy conditions to rebuild the sector with a much stronger emphasis on performance.

Even cities with good charter sectors could learn much from Baton Rouge’s wholistic approach.

Of course, the goal for Baton Rouge charters is not to achieve neutral effects.

But given the baseline, neutral is the first stop toward positive.

And given the rich entrepreneurial ecosystem that’s been built in the city, I’m optimistic that educational opportunity will continue to expand across the city.

Not just a great teacher. Not only a few good schools. An entire city.

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter and innovation schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

The results just came in for New Orleans.

Worries of Stagnation in New Orleans

Many observers, including myself, have been worried about the stagnating proficiency scores in New Orleans.

Screen Shot 2019-05-08 at 12.45.06 PM

One of the major problems of looking at proficiency cut-offs is they are a somewhat arbitrary cut-off line. For example, if a student started off extremely below proficient, but then improved to one test score question away from being proficient, the student would still show up as not proficient.

This is why Arnold Ventures funded the CREDO analysis. CREDO’s methodology allows us to examine achievement growth across all of the performance spectrum.

CREDO Analysis of New Orleans 

Fortunately for kids in New Orleans, CREDO found that New Orleans schools are continuing to outperform the state on academic growth.

Here is the most important slide from their analysis:

Screen Shot 2019-05-08 at 12.49.21 PM.png

The statistically positive achievements effects indicate that students in New Orleans are catching-up with the state somewhere in the range of .05 to .1 standard deviations a year.

To put these gains in some context, the black white achievement gap in the United States is .8 standard deviations.

These are strong, citywide annual gains. And it’s very important to emphasize that these are city level effects.

In most cities, the longer poor children stay in the system the further behind they get. In New Orleans, the opposite is true: the longer you enroll in New Orleans public schools, the closer you get to your peers across the state. 

This is not the case of a few charter schools doing well. This is about the whole city making gains.

The goal of city based education reform should be for all kids to benefit, not just a few.

And that’s happening in New Orleans.

Hopefully, these gains will continue to compound and, eventually, further raise citywide proficiency, graduation, and post-secondary success rates.

Is it ok that Lebron’s school chooses not to serve the most at-risk children in Akron?

I don’t have strong emotional attachments to athletes, but as far as athletes go, I like Lebron James.

Lebron got thrown into the spotlight as such young age, has accomplished amazing athletic feats, and is committed to trying to make his hometown better.

The New York Times recently wrote a piece on Lebron’s new public school in Akron.

Much could be written on how and why the school was covered, but I’ll stick to one point that I felt has not been covered: Lebron’s school refuses to serve students in the bottom 10% of performance.

From the article:

I Promise students were among those identified by the district as performing in the 10th to 25th percentile on their second-grade assessments. They were then admitted through a lottery.

Clearly, serving students in the 10th-25th percentile is not easy.

But serving kids in the 0-10th percentile is much more difficult.

A common critique of charter schools is that they don’t reach the very hardest to serve students.

There has not been a similar outcry against Lebron’s school.

Can you imagine the NYT headline if top charters had a formal policy about not serving students in the bottom 10% of performance?

___

On the substance of the issue, I’m sympathetic to Lebron’s approach. Serving students in the 10th-25th percentile well is both very hard and very important.

Also, starting without the hardest to reach kids helps reduce the risks of creating a new school.

I’ve seen new charter schools nearly collapse because of 5-10 kids who have severe mental health afflictions.

Of course, it’s not scalable for every school in a city to not serve the lowest performing 10% of students. And I also worry about segregating these students into separate schools.

But I think a reasonable policy would be to allow schools to get their footing and reach a bit of scale before having to build programs for the hardest to reach students.

 

Sometimes it doesn’t matter if it worked

Rand just release the initial evaluation of de Blasio’s Renewal School program.

The program cost $773 million.

The researchers found that the program did not improve student achievement.

The Renewal program has only been around for three years, so it feels a bit early to tell if it worked or not.

But I think “did it work?” is an important but secondary question to ask with a program like this.

The most important question is: “will it last?”

Even if the program had it gotten results, I’m skeptical that the program would have lasted.

The Renewal program had two major things going against it’s sustainability.

First, the program was deeply tied to de Blasio and the previous superintendent, Carmen Farina.

Second, there were no governance or legal protections for the program.

Taken together, this meant that the next mayor / superintendent would likely replace this initiative with their pet initiative.

This already had started to happen a year ago. When Farina stepped, the new superintendent, Richard Carranza, said the idea behind the Renewal program was “fuzzy.”

I viewed this as code for “it’s not my signature project.”

Admittedly, even I’m a bit surprised at how quickly the Renewal program collapsed. I thought it would at least live through the de Blasio administration.

We do live in a democracy (thankfully), so no government funded program, even if it works, is guaranteed to last forever.

But I would not have spent $773 million dollars on program that could be so easily undone.

I am such a deep believer in non-profit governance because it greatly increases the chance that something great can last.

In New Orleans, since the outset of the reforms, we’ve had three mayors and five superintendents.

And the work continues.

 

 

 

Minor feedback and reflections on Ray Dalio’s capitalism manifesto

Ray Dalio just published a two part manifesto on how capitalism needs to be reformed.

Dalio’s Diagnosis 

Dalio’s main point is that “capitalists don’t know how to divide the pie well and socialists don’t know how to growth the pie well.”

This is a problem for capitalist countries because rising inequality has both negative affects on individuals and countries: high inequality reduces economic mobility and weakens political governance.

I am not an expert in inequality, economic mobility, or governance, so it’s hard for me evaluate his specific claims.

But my instinct is too put a little less emphasis on capitalism.

Rising inequality seems to be the trend of almost all political systems.

Walter Schiedel’s book The Great Leveler does a good job of showing how inequality tends to rise in most societies (any system can be gamed) and that it’s often only war, revolutions, state collapse, and plagues that bring inequality back down.

Dalio thinks there’s something specifically inherent about capitalism that leads to inequality. I think there’s something generally inherent about governments and power.

Once humans left hunter gatherer societies, rising inequality has been the norm in most societies.

Capitalism is just the latest manifestation of the trend.

Stability vs. Growth 

Another argument which is somewhat implicit in Dalio’s argument but I wish would have been more explicit: given the incredible compounding of even small GDP increases, we should care much more about societal stability than maximizing growth.

Over the long-run, humans will be very rich by today’s standards so long as we can survive a couple more centuries.

In most cases, growth and stability go hand-in-hand, but when they don’t (perhaps in the case of inequality), we should err on the side of stability.

Can We Do Anything About It?

Schiedel’s book presents a bleak picture about a society’s ability to reduce inequality.

But humans haven’t been around for that long. We’ve never been this wealthy. And never before have so many humans lived in democratic societies.

It took us from the Industrial Revolution to the end of World War II to find a governmental system that fit our last major round technological and economic advancement.

Most advanced countries are now some form of capitalist democracies with large welfare states (at least by historical standards).

I hope our next governance search requires less bloodshed.

If it’s simply a matter of adjusting our current system, this might be possible.

If it’s a matter of finding whole new systems of governance, then that likely means wars and revolutions, which we should all fear given the weapons at our disposal.

Revisiting Diane Ravitch’s “A Challenge to KIPP”

7 years ago, Diane Ravitch wrote a blog post called “A Challenge to KIPP.”

In the piece, Diane accuses KIPP of cherry picking students and challenges them to serve an entire district.

When I gave my lecture, I chastised KIPP for encouraging the public perception that all charter schools are better than all public schools and for failing to denounce the growing numbers of incompetent, corrupt, and inept charter schools. I talked about the oft-heard complaint that KIPP cherry picks its students and has high attrition, which KIPP denies. I challenged KIPP to take over an entire inner city school district that was willing and show what it could do when no one was excluded.

She then makes the point in a more forceful way:

KIPP should find an impoverished district that is so desperate that it is willing to put all its students into KIPP’s care. Take them all: the children with disabilities, the children who don’t speak English, the children who are homeless, the children just released from the juvenile justice system,  the children who are angry and apathetic, and everyone else. No dumping. No selection. No cherry picking. Show us what you can do. Take them all.

This is part of what made New Orleans so important. KIPP didn’t grow to serve every kid in the city (which I don’t think would have been good, different kids thrive in different environments).

But charters did grow to serve every student in the city. And the last 20% of kids reached were harder to educate than the first 20% of kids reached.

But now, to use Diane’s words, charter schools in New Orleans “take them all.”

And they do it better than ever before.

Public charter schools in New Orleans have evolved to offer amazing programs for students with severe special needs. They also serve students in tough life circumstances, like teen mothers and those with extreme mental health and behavior diseases. None of these programs are perfect, but they are so much better than what New Orleans offered these students before, and some of these programs are on the path to be national exemplars.

And what about the results?

As readers of this blog know, New Orleans achieved some of the most impressive student achievement gains in our country’s recent history.

So is Diane Ravitch now saying: “New Orleans answered my challenge to KIPP. Charters enrolled all students and increased educational opportunity. I’m curious to learn more about how they did that. And I wonder if it could work in other cities?”

Of course not.

Unfortunately, with so many charter critics, the goalposts just continue to move.

But for the rest of us, we can keep the goalposts in place.

We can learn from successes like New Orleans, and we can try to figure out what’s scalable and what is not.

Beware of under powered studies

Mathematica just released a study on how charter middle schools impact college enrollment and completion. Before I dig into the study, some quick context.

The research base on the connection between test scores and life outcomes

An important research finding over the past decade has been that schools with neutral test score impacts sometimes achieve positive lifetime outcomes for their students.

Equally important, schools with positive test score impacts often don’t achieve positive life outcomes for their students.

Another important finding (a finding that is often glossed over by my libertarian leaning friends) is that schools with negative test score impacts are much less likely to achieve positive lifetime outcomes for their students.

Taken together, these findings have narrowed my belief in test based accountability. Previously, I put a lot of weight on the difference between positive, neutral, and negative test score schools.

Now I focus more attention on selectively transforming schools with negative test score impacts.

I also care much more about parent demand than I used to.

The Mathematica Study Limitations 

The Mathematica study has two very major limitations.

First, it covers charter schools from a previous era (2005-2007). In the subsequent decade, many charter schools (especially those we work closely with) have increased their focus on getting kids to and through college and into meaningful careers.

Second, the study is very small. The sample only included 10 urban charter schools. Additionally, only 3 schools in the entire study had poverty rates over 75%!

The small sample means the study is fairly under powered. With a sample this small, only large effects will be picked up. A modest but positive correlation between achievement and college graduation, for example, would not be statistically significant in this study. 

Other similar studies have looked at entire states, and included many more high poverty charter schools, so I place much less weight on this study.

Findings

The study did not find a statistically significant relationship between test score and college completion results. The chart below details the major findings, with each dot representing a school.

College Completion 

mathematica 2

When you eyeball the graph, some trends emerge, especially the low number of schools in the bottom right quadrant (high test effects, low college effects). 

When I emailed the authors about this trend, they said they were not statistically significant and we should be cautious to draw conclusions by looking at patterns the scatter plot. 

In Sum

The small sample size and the decade old study window period mean that the study itself won’t move my opinion on the issue that much.

Only 3 schools in the study were high poverty schools. Nearly all of the schools that we work with are high poverty schools.

In partnership with the Arnold Foundation, we’re supporting a bunch of more research on the issue, and I’m excited to learn more. 

I will change my mind if the findings change.  

 

 

Who benefits from neighborhood schools in Washington D.C.?

Screen Shot 2019-03-25 at 5.41.54 PM.png

Thomas Toch (a City Fund grantee) has a great piece on Washington D.C.’s public education system in the Washington Post Magazine.

The article tracks a few families through their process of finding public schools for their children.

A lot of progress has been made in D.C., and it was wonderful to read about this progress through lens of families trying to find a great public school. Much of our work is strategically abstract, but there’s nothing abstract about a family trying to find a school.

I strongly encourage you to read the whole thing. It will likely be one of the best pieces you read this year about what public education could be.

I’ll just highlight one important point from the article:

A so-called in-boundary preference makes it particularly tough to get into the city’s best traditional neighborhood schools. Carving the city into attendance zones and guaranteeing students a spot in the traditional public school in their zone is a way of reconciling school choice with the right of every student to a free public education.

But not surprisingly, families living within the boundaries of top schools exercise their in-boundary advantage more often than families with weak neighborhood options. And because many of the highest-performing neighborhood schools are in predominantly white and more-affluent sections of the city, the My School DC preferences weaken the system’s ability to reduce long-standing racial and economic segregation in Washington’s public schools, despite citywide school choice. For the same reason, undoing the preferences would be politically impossible.

Overall, only 27% of families choose to attend their neighborhood school. Those that do tend to be white and living in wealthy neighborhoods.

Additionally, approximately 50% of DC’s white families send their children to private schools.

So wealthy, white families use exclusive neighborhood schools and private schools to educate their children, but somehow it’s charter schools that serve low-income families that are the problem?

This makes no sense. At some point, people will realize this. I just worry about how long it might take.

 

Be curious about the strengths of flawed people

When I was younger I would focus on the weaknesses of successful people.

I would think: “this person is very successful, and it’s clear to me they are doing X wrong, so if I do X right I’ll be even more successful than they are.”

This is a wrongheaded way to think about things and it’s taken me years to adjust my mental model.

Now when I meet a successful person I think: “this person is very successful, and it’s clear to me they are doing X wrong, so how are they still so successful?”

This is a small shift in a mental model that has had big impact in how I think.

A few takeaways from having made this shift:

I find it very pleasant to spend more time thinking about why people are amazing rather than why people are flawed. This is not a reason in and of itself to think this way, but it’s a nice bonus.

It’s clear that being very very good at a few things that are aligned with one’s role can lead to tremendous output. Specialization rules the day.

It’s clear that being very very good at a few ways of thinking and behaviors can be lead to tremendous output even if coupled with major flaws in ways of thinking and behavior.

A common flaw I see in successful people is that they are just as stubborn about the things they know a lot about as they are about the things they know a little about. This is a major flaw that has little repercussions so long as they stay specialized. But it can blow up in their faces if they branch out. This is a major risk in philanthropy.

Anyways, I suggest you give this way of thinking a try. Anytime you meet someone who’s done something amazing, try to be curious about why they were able to accomplish amazing things despite all their flaws.

The low ROI of focusing narrowly on teacher quality

Eric A. Hanushek, Marc Piopiunik, and Simon Wiederhold just published a study on international teacher quality.

They looked at an international sample of teacher cognitive ability (as measured by OECD’s Program for the International Assessment of Adult Competencies) and student achievement (as measured by PISA scores).

They found a lot of variation of teacher cognitive quality across countries.

Teachers in Finland and Japan score very well. Teachers in Italy, Russia, and Israel score poorly.

After running regressions and controls, the authors estimate that a one standard deviation increase in teacher quality leads to a .15 standard deviation in student achievement.

To put a one standard deviation increase in context, a one standard deviation bump from the mean in IQ scores is about 15 points.

I can’t think of a realistic intervention that, at scale, will lead to that kind of increase in teacher cognitive ability across the entire United States.

For me, this is just one more study that validates the idea that institutional change is the best hope we have for increasing educational opportunity.

High-performing urban charter school sectors across the United States often achieve a ~.1 standard deviation effect over the course of three to four years.

I believe we can thoughtfully grow these non-profit public schools in a manner that maintains quality at scale. I also think we can empower amazing district leaders by letting them adopt non-profit governance for their schools.

I really can’t think of a way that we increase teacher cognitive ability by one standard deviation at scale.

In every city we work in, we do invest in recruiting and supporting great teachers. Teachers should be recruited hard, paid more, and supported better.

But I’m skeptical that focusing most of our energy on improving our teacher pool is the best way to help low-income kids.

Instead, I think we should focus much of our energy on creating and expanding great public institutions where teachers can work.

Another way to say it: I don’t blame our teachers for the dysfunction that plagues many schools in cities across the United States.

It’s their employers that should be the focus of our attention.