Stanford researchers find that New Orleans and Chicago are doing amazing things for black children

The New York Times / Upshot just took a massive Stanford researcher database and turned it into an easy to use webpage.

This was a big undertaking and these types of projects are some of the best of modern journalism. Kudos to them.

The research methodology is great in its scope (every city in America) though rather crude in its precision (raw proficiency gains from 2009 to 2015). It’s not as accurate as experimental and quasi-experimental research, but it’s still useful in taking a broad look across the country.

Which of the largest 200 school districts in the country saw the most growth?

Here’s the top 10:

Screen Shot 2017-12-06 at 7.00.57 AM

In 5 years, all of these districts achieved at least 5.5 years of academic growth. Of the 200 largest school districts in the nation, these are the highest performers.

Are these cities very similar to each other? 

Not at all. And this is where I think the reporters could have done much more. The New York Times journalists rightfully pointed out Chicago as an outlier, but they did little else to tease out the vast differences between these high growth districts.

Let’s take a look at these top growth districts by free and reduced lunch rates:

FRL3

See any differences?

Only four districts – Salem Keizer, Garden Grove, Chicago, and Orleans – have students bodies where the majority of students are economic disadvantaged.

And Chicago and Orleans are in a category of their own, with 80%+ students receiving free and reduced lunch during the years of the study. The task these districts face is 100x more difficult than that of the low poverty school districts.

Here’s the district African-American student enrollment percentages:

AA rate

See any differences?

None of the highest growth districts in the country except for Chicago and Orleans serve many African-American students.

New Orleans and Chicago serve many students whose lives are still shaped by the deep, generational poverty that is rooted in our country’s horrific history of slavery.

The fact that their academic growth is amongst the very best in the nation should be a huge cause for celebration and hope.

What we talk about when we talk about high growth districts

It took me about 45 min to create the above charts. I did quick google searches so the numbers probably aren’t perfect, but they paint a pretty clear picture.

Yet in the 10+ articles I’ve read about the Stanford research data set not one has mentioned that New Orleans is the only majority black school district to be in the top ten growth school districts in the nation. 

And in case you’re wondering if the New Orleans data includes all of its schools, it does. I emailed the researchers and they confirmed that the data includes all district and charter schools in the city boundaries.

So let’s give a big shout out to the amazing educators, students, and families who achieved unprecedented gains in learning in New Orleans between 2009 and 2015.

The New York Times might not have noticed, but hopefully others eventually will.

Does anyone know why Chicago children are getting smarter?

If you just read the headlines, you might suspect that Chicago’s public schools are in a terrible tailspin. Part of this is the noise of big city politics. Part of this stems from cloud of violence that hangs over the city.

But Chicago has improved on academic test scores more than most other cities in the country. Rather than one of the worst, Chicago is one of the best.

A recent report by Sean Reardon and Rebecca Hinze-Pifer found that between 2009 and 2014:

“This [student achievement] growth rate [of Chicago] is higher than 96% of all districts in the US. Among the 100 largest districts in the country, the average growth rate from third to eighth grade is 0.95 grade equivalents per year; Chicago has the highest growth rate between third and eighth grade of any large district in the United States.”

The authors admit that they don’t know why this occurred.

I can’t prove why Chicago kids are getting smarter, but I have a hypothesis.

What’s Been Going on in Chicago Public Schools? 

One way to try and solve the mystery of why Chicago children are getting smarter is to look at the district’s previous major initiatives.

As this report details, between roughy 1990 and 2010 there were three overarching eras of reform in Chicago: the decentralization era, the the accountability era, and the do a lot of things era.

The authors are very careful to not attribute a causal relationship between reform eras and outcomes. The reforms were messy and not rolled out in an experimental manner – so fair enough.

But in this post I’ll try and make my best guess on what was causal and what was not.

The Decentralization Era

The decentralization era was best known for the creation of Local School Councils. This reform gave local councils real control over decisions about how schools were run. The councils were made up of school leadership, parents, and community members.

The councils always seemed like a terrible idea to me. It’s basically taking all we know about charter schools (good central offices, scalable instructional programs, governance matters) and doing the exact opposite!

Not surprisingly, research on the reforms found that the councils had some positive effects on advantage communities, but were least likely to improve schools in low-income communites. Communities with low social capital didn’t gain a lot from ad-hoc and poorly constructed local boards.

I’m very skeptical that the decentralization era and school councils were the root cause of later gains.

The Accountability Era

In 1995 Mayor Daly put in Paul Valls as the superintendent (I later worked with Paul when he was the superintendent of the RSD in Louisiana). Vallas, who did not have deep instructional expertise, used test driven accountability to try and make things better.

New tests, promotional standards, and interventions for failing schools were all put in place.

The reforms had better impacts for low-performing schools; the researchers noted:

“This was the only era to show large improvements in the lowest- achieving schools. However, the patterns in test scores in the lowest-performing schools suggest that some of the improvements resulted from instruction that was aligned specifically to the high stakes tests.”

This matches other research on accountability reforms: you tend to see gains in the lowest preforming schools, but the high stakes can cause narrowing of the curriculum.

 

The Do a Lot of Things Era

Arne Duncan came in after Vallas, and he instituted a lot of reforms.

Arne launched 100 new schools, implemented internal district instructional and curricular reforms, overhauled school leadership pipelines, and placed a deep focus on on-time high school progression.

Perhaps the biggest initiative of this era was the Renaissance 2010 project, which launched about a 100 new district, charter, and contract schools between 2005 and 2010.

Unfortunately, no one has conducted a full evaluation of the program. Someone should do this!

Two interim research reports came out around 2010. One study, which only included a few years of data from the early Renaissance cohorts, found that the new schools performed about the same as the existing district schools. The other study was inconclusive.

Not much help from the research community.

A lot of work was also done on school leadership. The Chicago Public Education Fund, in partnership with the district, invested heavily in school leader development, placing bets on both district based and non-profit providers.

The lastest research I could find on these programs found that “results indicate that one-year learning gains in elementary and high schools led by Fund-supported principals were not different than those in other similar schools.”

Another major reform, another mediocre result.

All told, researchers found that this era produced more gains in high school than elementary schools, but wrote: “while the effects of the dominant policies of Eras 1 and 2 are largely understood, much research remains to be done to understand both the positive and problematic effects of the policies in Era 3.”

Not super helpful, especially since this is the era that preceded the large gains in test scores that occurred after 2009.

What About the Charter Sector? 

CREDO published a report on Chicago charters that covered test scores from 2010 to 2012, which is right in the middle of the period where Chicago saw a lot of gains.

The study found +.01 effects in reading and +.03 effects in math. These effects amount to about a month or so of extra learning per year, maybe a bit less. Given Chicago’s relatively small charter market share, and the modest size of these positive effects, it’s unlikely that charters themselves accounted for the 2009-2014 gains.

A more recent study, which just looked at charter high school performance from 2010-2013, found much larger effects: +.2 effects on ACT related tests and much higher college enrollment rates.

These are large effects, but they are for high school only. The study lauding Chicago’s gains only covered grades 3-8.

So WTF Happened in Chicago to Make Kids Smarter?

To summarize: Chicago improved its test scores more than any other big city in the country, and researchers really don’t know why.

So why are Chicago kids getting smarter?

Here’s my guess: competition and accountability lifted all boats.

When you put accountability in place (the Vallas era) and then launch a 100 new schools (the Duncan era) you get a city where school leaders know there are consequences for failure and the best of the new schools begin to raise the bar for what’s possible.

This theory helps explain why the Renaissance schools and charter effects were a bit muted. In the studies on these reforms, researchers compared the new schools to existing schools. So if the existing schools were improving due to increased competition, you would not see large relative effects for the new schools.

I can’t prove that accountability and competition caused the results, but in many sectors accountability and competition make everyone better. It also fits stories we’ve seen elsewhere. In place like Denver and Washington D.C. increased competition led to all boats rising in the public school system.

If you have a better theory, let me know.

What Should Chicago Do Now? 

Here’s another tough question: if it was accountability and competition that caused Chicago’s gains, how should this impact Chicago’s future strategy?

Since 2002 (while the district was getting much better!) Chicago enrollment plummeted from 440,000 students to 370,000 students.

This means that there are lot of under-enrolled schools in the district and the city might have to go through another round of painful closures.

This also it means it’s harder to push the very reform (opening new schools) that might have driven Chicago’s previous gains in achievement.

So what should the city do?

Reasonable people can surely disagree, but I would continue to create new schools, albeit in a different fashion.

First, I’d open new schools in the areas where population is increasing. Chicago is made up of a lot of neighborhoods, and not all neighborhoods are losing children.

Second, I would do some replacement work. Instead of closing all the under-enrolled schools, I’d try and select some neighborhoods where there’s enough child density that you could imagine families coming back to the public schools if there were better options. I’d launch replacement schools in these neighborhoods.

There are clear drawbacks to this strategy. Politically, it’s hard to justify opening schools when you’re in the midst of closures. Programmatically, it’s hard sell to get the operators of new schools to open up in neighbors with shrinking enrollment.

But I think it’s the best thing for children.

Lastly, I might also try and launch some diverse by design schools.

In a city as diverse as Chicago, it’s sad that it’s schools are so segregated.

The Last Word

Chicago’s Chief Education Officer, Janice Jackson, recently gave her take on why things are better.

Her list: pre-k, better professional development, better curriculum, competition from private and charter schools, and clear accountability standards.

In her own words:

“I believe the level of transparency we have provided around what a quality school is has been transformational in this district.”

Is San Francisco progressive?

San Francisco is a thriving city and I love living here. It nearly has it all: great people, a vibrant economy, and natural beauty in all directions.

But there is one thing San Francisco does’t have: a public education system that does right by low-income African American and Latino students.

One of the wealthiest and most innovative cities in the world can’t teach low-income kids to read, write, and do math.

Nearly every other city in California performs better than San Francisco in educating low-income students, and it’s not like most of the cities are knocking it out of the park.

And the worst part of it all is that the fix is not too hard. Within a decade, I bet San Francisco could be one of the highest performing urban school systems in the country.

But I’m skeptical that will occur. The people who control the system do not want it to change.

Traditional Schools Harm Low-Income Students in San Francisco 

San Francisco is one of the richest cities in the world. But when it comes to serving low-income African-American students, the city is trounced by other cities across the state.

recent report from Innovate Public Schools provides the details.

Screen Shot 2017-11-10 at 8.32.36 PM

Low income Latino students don’t do much better.

Screen Shot 2017-11-10 at 8.33.54 PM

Charter Schools Help Low-Income Students in San Francisco 

Is there something about low-income kids in San Francisco that makes them impossible to educate?

No.

Charter schools in San Francisco are doing a much better job educating low-income students.

Screen Shot 2017-11-10 at 9.05.51 PM

More rigorous research (quasi-experimental methadology that controls for socioeconomic variables) has also found that charter schools in the Bay Area dramatically outperform the traditional system. This study found positive .2 standard deviation annual effects in math and .1 effects in reading.

It Could Get Better But It Probably Won’t

Cities like Denver, Washington D.C., and New Orleans are proving that there’s a better way to do public education.

Denver is continuing to deliver results over ten years after its reforms began. Washington D.C. has seen its scores skyrocket on the Nation’s Assessment of Educational Progress. New Orleans achieved amongst the greatest educational gains that the nation has recently seen.

Our team has the privilege of working with cities to make their educational systems better, but I spend most of my time on the road because the San Francisco Unified School Board doesn’t have any interest in the reforms that could make the city’s public schools great.

Unlike the aforementioned cities, San Francisco Unified is extremely unwelcoming to high-quality charter schools.

Instead of partnering with high-quality non-profit charter schools, the school district continues to insist that it will fix itself.

But this will not happen. And low-income kids will keep getting screwed.

So no, San Francisco is not progressive. At least not when it comes to public education.

SaveSaveSaveSaveSaveSave

Don’t sacrifice the truth about charter schools in order to be agreeable

The New York Times just wrote a positive editorial about charter schools.

The editorial opened with this sentence [emphasis mine]:

“New York City is one of the rare places in the country where charter schools generally have made good on the promise to outperform conventional public schools in exchange for flexibility from the state that lets them lengthen the school day, alter the curriculum, do away with tenure and change how teachers are compensated.”

As a reminder, here’s the average effect of urban charter schools – from a study by the same researchers that the New York Times linked to in the above lead!

Screen Shot 2017-06-15 at 8.51.23 AM

So why did the New York Times write such a factually incorrect lead?

I think they probably did it to disarm those who might oppose them. By saying that most charter sectors have failed, they are aligning themselves with those suspicious of charter schools, which perhaps increases their ability to influence those on the fence.

This is a bad tactic. And it’s a habit I’ve been trying to kick: I too sometimes publicly hedge on the actual facts in order to relate to an audience.

This type of hedging is doubly dangerous.

First, it prizes short-term affiliation over the truth, which will eventually reduce your credibility when people find out what you really believe.

Second, you risk starting to believe yourself. It’s very difficult to maintain thoughtful and evidence driven policies under the best of circumstances. If you consistently say things you don’t really believe, you’ll soon forget what you really believe.

Here’s a better tactic: be expressive about values while you’re being direct about your beliefs.

Constantly talk about why you care about children, poverty, and the future of our country – at the same time you defend policies (like charter schools) that are controversial but impactful.

It’s good practice to expand the tent through shared values.

But don’t trade the truth for agreeableness.

It’s dishonest and counterproductive.

A thoughtful response from Chris Cerf

Chris Cerf, the current superintendent of Newark and the former state superintendent of New Jersey, posted a thoughtful response to my post on the Newark Harvard study.

A couple of thoughts.

First, I’ve had the opportunity to interact with Chris on and off over the past decade, and I have immense respect for his heart and mind. Kids in New Jersey are better off because of his leadership.

Second, I appreciated the tenor of Chris’ post. A primary reason I write this blog is so that  people in the education reform family can have public disagreements and learn from each other. Chris’ tone and use of data helped me get smarter on Newark.

While there is some cost to this approach (those who oppose our work can try to exploit our disagreements), ultimately, I think the gain of learning through public debate is well worth it.

Over the long haul, our success will have more to do on whether or not we continually delivered great educational opportunities for children than whether or not we win or lose the PR fire of the week. And putting our ideas out there for public testing is a good way to get smarter on how we deliver better opportunities for children.

As for the substance of Chris’ points, you should read the piece for yourself, but I found the below two graphs to be useful.

The first compares Newark’s overall performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s relative performance to similarly situated districts has improved greatly over the past seven years.

newark-graph1

The second compares Newark’s traditional school performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s traditional school performance was fairly flat until 2014, but has grown rapidly since then.

newark-graph2

I’m not sure that Chris and I disagree on the data story.

Both of us, I think, would say that the early gains in Newark were driven by the open / close / shift strategy.

As for the improve the traditional sector strategy, Chris points to the last few years of growth to demonstrate that the reforms are starting to deliver for all schools – and that now that the foundation has been set, these gains will likely continue.

I emphasized the flat results for the first four years of the district improvement strategy and wondered whether the improve strategy was worth it – or whether putting more resources into the open / close / shift strategy would have laid an even better foundation for long-term gains.

It’s a great question to grapple with.

Lastly, our interpretations of the study go to an even more fundamental question of how we measure success: should a city’s reform efforts be evaluated on the cumulative gains it achieved during the transformation process, or should it be evaluated on the gains being delivered once a city is through a transformation process?

When we partner with a city, our team holds ourselves for cumulative gains over the initial 5-10 years of reform, but perhaps a more important metric is whether a city achieves a new and better equilibrium by the end of the reforms.

Chris has pushed me to think hard about this question, our team will be smarter for it.

Could Newark have achieved more?

Harvard researchers just published a study on Newark.

In large part due to Mark Zuckerberg’s $100 million gift to the reform efforts, the city has seen a lot of coverage, including the well done book The Prize.

Most of this coverage has been negative.

The Harvard Study: Teasing Out Annual Effects From 2010-2016

The Harvard study found that the Newark reforms, in the most recent year of the study, had a positive impact on ELA (.07 SD) and no impact on math. In the early years of the reforms, the achievement effects were negative.

Unfortunately, the researchers only looked at annual effects (were there positive or negative effects in any given year) rather than calculating cumulative impacts (what was the total impact over these years). Because of this, we can’t perfectly compare the Newark results with results of other city based studies, such as Doug Harris’ evaluations of the New Orleans efforts.

What Caused the Positive Effects? 

While the authors didn’t calculate the cumulative effects of the reforms, they did do something wonderful.

The authors separated out the effects of two different strategies: (1) improving existing schools vs. (2) expanding high-performing schools, closing low-performing schools, and facilitating the transfer of students out of low-performing schools and into high-performing schools.

The improving existing schools approach included replacing large numbers of principals, renegotiating the union contract, implementing new data systems, and extending learning time.

The open / close / shift enrollment approach included adding to the enrollment of high-performing charter and district schools, closing underperforming schools (11 traditional schools and 3 charters), and implementing a unified enrollment system that made choosing schools much easier for families. These reforms increased charter enrollment from 14% to 28%.

The results are striking.

In math, the improve strategy achieved a .08 negative effect in math while the open / close / shift strategy had a positive .04 effect.

In ELA both strategies had positive effects, but the open / close / shift was responsible for 62% of the overall positive effect.

Perhaps most importantly: the open / close / shift strategy achieved positive effects in every year of the study. 

Opening and closing schools, and shifting student enrollment, increased student learning.

Could Newark have Achieved New Orleans’ Level Gains?

In roughly the same number of years as the Newark reforms, New Orleans achieved positive .2-.4 standard deviation effects.

New Orleans, of course, relied much more on the open / close / shift than it did on improvement.

As noted above, we can’t do a direct apples-to-apples comparison of the studies. But I’d be surprised if Newark’s cumulative effect is as large as the New Orleans effect given that Newark saw negative effects in 3 of the 5 years of reforms.

We’ll never know if Newark could have achieved similar gains to New Orleans, but in many ways the Newark circumstances were more enviable than those of New Orleans.

Newark is home to one of the highest-performing charter sectors in the country, while New Orleans had no large effective charter operators in the city right after Katrina.

Newark’s initial philanthropic investment was much larger than that received in New Orleans after Katrina. And the public per-pupil in Newark is about 3x that of New Orleans.

Newark students also didn’t have to go through the immense trauma of Katrina.

That being said: perhaps New Orleans style reforms would have not worked in Newark.

Maybe the politics would not have allowed for it… though it’s worth remembering that Shavar Jeffries, now the CEO of Democrats for Education Reform, was only about 2,000 votes away from beating Ras Baraka in the 2014 Newark mayoral race.

Or perhaps the charter sector would not have been able to grow much faster than it did; notably, charter effects did decline as they enrolled more students… though perhaps more charter capacity would have emerged under better conditions.

All told, my guess is that Newark did not increase educational opportunity as much as it could have.

A 2010 Prediction 

On this blog, I try to be honest about both my mistakes and successes.

It would be disingenuous for me to write a post arguing that, in 2010, no one could have predicted which strategy would have been better.

The fact is that I did predict this. In 2010, when it became clear that there was going to be major reforms in Newark, I created a powerpoint deck to give advice to national educational leaders. In the deck, I modeled out how quickly I thought you could expand great schools and close failing schools.

Given the relatively small size of Newark public schools (50,000 students), as well as the number of high-quality charter schools, I felt confident that the vast majority of Newark students could attend a high-quality school within ten years.

This ended up being the strategy we took in New Orleans.

It was not the strategy taken in Newark.

Learning 

I have a lot of love for the philanthropists, political leaders, and educators who spent the good part of a decade trying to make things better for kids. I know many of them well and have had nothing but positive interactions.

In terms of rigorous third party research results, we also knew a lot less in 2010 than we do now. There have been a wave of recent studies documenting the positive effects of expanding the best schools and closing the worst.

And while we still suffer from a dearth of research on citywide turnaround efforts, outside of the early results in Lawrence, Massachusetts, we really haven’t seen a citywide turnaround reform effort significantly raise achievement without a focus on opening and closing schools (research has shown that Washington D.C. improved their public schools, but the city also leaned heavily on opening and closing schools).

So perhaps the better question is not what should have Newark leaders done in 2010, but what should similarly situated civic leaders do in 2017?

Civic leaders should embrace strategies that maximize the expansion of the best schools and close the worst schools.

Of course this need not be the only focus of a city turnaround effort, but the Newark study has increased my belief that it should be the primary focus.

Book Review: Ray Dalio’s Principles

dalioo

How to read it

I just finished Ray Dalio’s Principles and found the book to be both uneven and very useful.

Sometimes I’d read a few paragraphs and think “I should probably spend the next decade trying to get that right” and sometimes I’d read 10 pages and think “that’s a long way of saying something that everyone already knows.”

Thanks to Ken Bubp, our team is also digging into the book together, which I hope will make us better.

Already, there are a couple of areas where the book made me want to get much better – a few of those detailed below.

The point of a work conversations is to find the truth

The point of social conversation is not  to find the truth: it’s to build bonds, have fun, and play status games.

These social habits tend to bleed into work conversations and it takes a lot of deliberate effort to create an environment where truth seeking is the paramount goal of a conversation.

Specifically, truth finding conversations require the implementation of two difficult to reconcile mindsets: not holding tight too tight to one’s own opinions + being aggressive enough to pressure test another person’s opinions.

So team members have to constantly toggle between:

you’re a smart person who shares my values so I need to figure out why you think I’m wrong because I might be wrong

and

I think you’re wrong because of X and I will not back down from this opinion for the sake of being agreeable

This is tough to do.

Normalize and embrace emotional and intellectual pain

In social relationships, we tend to avoid pain. Doing so is often bad for the long-term health of social relationships, and it can be devastating in professional settings.

Too often in work:

  1. It is considered harmful to cause someone else intellectual or emotional pain.
  2. The first reaction of feeling pain is often to attack others and build self-promoting narratives.

Both of these are very hard to unwind.

It’s very difficult to create a cultural where it is ok to cause someone the right kind of pain.

And it’s very hard to have the individual fortitude to build an internal monologue that goes like this: “the fact that I’m feeling a lot of pain means this is a prime opportunity for growth.”

With fluff and abuse at the polar ends of the cultural spectrum, getting this right is hard.

Leaders are defenders of the standard 

The standard for thinking and performance starts at the top, and the leadership of an organization must constantly live the standard and hold all others to the standard (this is also the principle of the great book The Score Takes Care of Itself).

It takes a lot of mental stamina to hold the bar: in 1-1 check-ins, in team meetings, in external conversations with partners…

…to never nod your head unless you mean it is flat out exhausting… to always ask the next probing question is annoying…. to say it’s not good enough yet is tiring…

But if you don’t do it you, your team, your organization will slide into mediocrity.

Getting the balance right between people, processes, and technology

Leaders have a finite amount of energy and capital to allocate across people, processes, and technology.

Different organizations, industries, and time periods require different allocations – as well as different types of expenditures.

Dalio goes deep in how he managed these as he grew his hedge fund, early on relying more on technology than most of his peers. While some his efforts are generalizable, many of them are not.

But the discipline by which he came to his approach is generalizable, and it’s something I want to keep working on as our team, technology, and industry evolve.

Application

After our team conversation, we’ll have to figure out how to apply these lessons. In some cases, it might simply be through consciously applying these principles in our conversations; in some cases, we’ll build processes; and perhaps down the road we’ll build out some technology.

SaveSave