Category Archives: Hard Things

Does anyone know why Chicago children are getting smarter?

If you just read the headlines, you might suspect that Chicago’s public schools are in a terrible tailspin. Part of this is the noise of big city politics. Part of this stems from cloud of violence that hangs over the city.

But Chicago has improved on academic test scores more than most other cities in the country. Rather than one of the worst, Chicago is one of the best.

A recent report by Sean Reardon and Rebecca Hinze-Pifer found that between 2009 and 2014:

“This [student achievement] growth rate [of Chicago] is higher than 96% of all districts in the US. Among the 100 largest districts in the country, the average growth rate from third to eighth grade is 0.95 grade equivalents per year; Chicago has the highest growth rate between third and eighth grade of any large district in the United States.”

The authors admit that they don’t know why this occurred.

I can’t prove why Chicago kids are getting smarter, but I have a hypothesis.

What’s Been Going on in Chicago Public Schools? 

One way to try and solve the mystery of why Chicago children are getting smarter is to look at the district’s previous major initiatives.

As this report details, between roughy 1990 and 2010 there were three overarching eras of reform in Chicago: the decentralization era, the the accountability era, and the do a lot of things era.

The authors are very careful to not attribute a causal relationship between reform eras and outcomes. The reforms were messy and not rolled out in an experimental manner – so fair enough.

But in this post I’ll try and make my best guess on what was causal and what was not.

The Decentralization Era

The decentralization era was best known for the creation of Local School Councils. This reform gave local councils real control over decisions about how schools were run. The councils were made up of school leadership, parents, and community members.

The councils always seemed like a terrible idea to me. It’s basically taking all we know about charter schools (good central offices, scalable instructional programs, governance matters) and doing the exact opposite!

Not surprisingly, research on the reforms found that the councils had some positive effects on advantage communities, but were least likely to improve schools in low-income communites. Communities with low social capital didn’t gain a lot from ad-hoc and poorly constructed local boards.

I’m very skeptical that the decentralization era and school councils were the root cause of later gains.

The Accountability Era

In 1995 Mayor Daly put in Paul Valls as the superintendent (I later worked with Paul when he was the superintendent of the RSD in Louisiana). Vallas, who did not have deep instructional expertise, used test driven accountability to try and make things better.

New tests, promotional standards, and interventions for failing schools were all put in place.

The reforms had better impacts for low-performing schools; the researchers noted:

“This was the only era to show large improvements in the lowest- achieving schools. However, the patterns in test scores in the lowest-performing schools suggest that some of the improvements resulted from instruction that was aligned specifically to the high stakes tests.”

This matches other research on accountability reforms: you tend to see gains in the lowest preforming schools, but the high stakes can cause narrowing of the curriculum.

 

The Do a Lot of Things Era

Arne Duncan came in after Vallas, and he instituted a lot of reforms.

Arne launched 100 new schools, implemented internal district instructional and curricular reforms, overhauled school leadership pipelines, and placed a deep focus on on-time high school progression.

Perhaps the biggest initiative of this era was the Renaissance 2010 project, which launched about a 100 new district, charter, and contract schools between 2005 and 2010.

Unfortunately, no one has conducted a full evaluation of the program. Someone should do this!

Two interim research reports came out around 2010. One study, which only included a few years of data from the early Renaissance cohorts, found that the new schools performed about the same as the existing district schools. The other study was inconclusive.

Not much help from the research community.

A lot of work was also done on school leadership. The Chicago Public Education Fund, in partnership with the district, invested heavily in school leader development, placing bets on both district based and non-profit providers.

The lastest research I could find on these programs found that “results indicate that one-year learning gains in elementary and high schools led by Fund-supported principals were not different than those in other similar schools.”

Another major reform, another mediocre result.

All told, researchers found that this era produced more gains in high school than elementary schools, but wrote: “while the effects of the dominant policies of Eras 1 and 2 are largely understood, much research remains to be done to understand both the positive and problematic effects of the policies in Era 3.”

Not super helpful, especially since this is the era that preceded the large gains in test scores that occurred after 2009.

What About the Charter Sector? 

CREDO published a report on Chicago charters that covered test scores from 2010 to 2012, which is right in the middle of the period where Chicago saw a lot of gains.

The study found +.01 effects in reading and +.03 effects in math. These effects amount to about a month or so of extra learning per year, maybe a bit less. Given Chicago’s relatively small charter market share, and the modest size of these positive effects, it’s unlikely that charters themselves accounted for the 2009-2014 gains.

A more recent study, which just looked at charter high school performance from 2010-2013, found much larger effects: +.2 effects on ACT related tests and much higher college enrollment rates.

These are large effects, but they are for high school only. The study lauding Chicago’s gains only covered grades 3-8.

So WTF Happened in Chicago to Make Kids Smarter?

To summarize: Chicago improved its test scores more than any other big city in the country, and researchers really don’t know why.

So why are Chicago kids getting smarter?

Here’s my guess: competition and accountability lifted all boats.

When you put accountability in place (the Vallas era) and then launch a 100 new schools (the Duncan era) you get a city where school leaders know there are consequences for failure and the best of the new schools begin to raise the bar for what’s possible.

This theory helps explain why the Renaissance schools and charter effects were a bit muted. In the studies on these reforms, researchers compared the new schools to existing schools. So if the existing schools were improving due to increased competition, you would not see large relative effects for the new schools.

I can’t prove that accountability and competition caused the results, but in many sectors accountability and competition make everyone better. It also fits stories we’ve seen elsewhere. In place like Denver and Washington D.C. increased competition led to all boats rising in the public school system.

If you have a better theory, let me know.

What Should Chicago Do Now? 

Here’s another tough question: if it was accountability and competition that caused Chicago’s gains, how should this impact Chicago’s future strategy?

Since 2002 (while the district was getting much better!) Chicago enrollment plummeted from 440,000 students to 370,000 students.

This means that there are lot of under-enrolled schools in the district and the city might have to go through another round of painful closures.

This also it means it’s harder to push the very reform (opening new schools) that might have driven Chicago’s previous gains in achievement.

So what should the city do?

Reasonable people can surely disagree, but I would continue to create new schools, albeit in a different fashion.

First, I’d open new schools in the areas where population is increasing. Chicago is made up of a lot of neighborhoods, and not all neighborhoods are losing children.

Second, I would do some replacement work. Instead of closing all the under-enrolled schools, I’d try and select some neighborhoods where there’s enough child density that you could imagine families coming back to the public schools if there were better options. I’d launch replacement schools in these neighborhoods.

There are clear drawbacks to this strategy. Politically, it’s hard to justify opening schools when you’re in the midst of closures. Programmatically, it’s hard sell to get the operators of new schools to open up in neighbors with shrinking enrollment.

But I think it’s the best thing for children.

Lastly, I might also try and launch some diverse by design schools.

In a city as diverse as Chicago, it’s sad that it’s schools are so segregated.

The Last Word

Chicago’s Chief Education Officer, Janice Jackson, recently gave her take on why things are better.

Her list: pre-k, better professional development, better curriculum, competition from private and charter schools, and clear accountability standards.

In her own words:

“I believe the level of transparency we have provided around what a quality school is has been transformational in this district.”

A thoughtful response from Chris Cerf

Chris Cerf, the current superintendent of Newark and the former state superintendent of New Jersey, posted a thoughtful response to my post on the Newark Harvard study.

A couple of thoughts.

First, I’ve had the opportunity to interact with Chris on and off over the past decade, and I have immense respect for his heart and mind. Kids in New Jersey are better off because of his leadership.

Second, I appreciated the tenor of Chris’ post. A primary reason I write this blog is so that  people in the education reform family can have public disagreements and learn from each other. Chris’ tone and use of data helped me get smarter on Newark.

While there is some cost to this approach (those who oppose our work can try to exploit our disagreements), ultimately, I think the gain of learning through public debate is well worth it.

Over the long haul, our success will have more to do on whether or not we continually delivered great educational opportunities for children than whether or not we win or lose the PR fire of the week. And putting our ideas out there for public testing is a good way to get smarter on how we deliver better opportunities for children.

As for the substance of Chris’ points, you should read the piece for yourself, but I found the below two graphs to be useful.

The first compares Newark’s overall performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s relative performance to similarly situated districts has improved greatly over the past seven years.

newark-graph1

The second compares Newark’s traditional school performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s traditional school performance was fairly flat until 2014, but has grown rapidly since then.

newark-graph2

I’m not sure that Chris and I disagree on the data story.

Both of us, I think, would say that the early gains in Newark were driven by the open / close / shift strategy.

As for the improve the traditional sector strategy, Chris points to the last few years of growth to demonstrate that the reforms are starting to deliver for all schools – and that now that the foundation has been set, these gains will likely continue.

I emphasized the flat results for the first four years of the district improvement strategy and wondered whether the improve strategy was worth it – or whether putting more resources into the open / close / shift strategy would have laid an even better foundation for long-term gains.

It’s a great question to grapple with.

Lastly, our interpretations of the study go to an even more fundamental question of how we measure success: should a city’s reform efforts be evaluated on the cumulative gains it achieved during the transformation process, or should it be evaluated on the gains being delivered once a city is through a transformation process?

When we partner with a city, our team holds ourselves for cumulative gains over the initial 5-10 years of reform, but perhaps a more important metric is whether a city achieves a new and better equilibrium by the end of the reforms.

Chris has pushed me to think hard about this question, our team will be smarter for it.

Book Review: Ray Dalio’s Principles

dalioo

How to read it

I just finished Ray Dalio’s Principles and found the book to be both uneven and very useful.

Sometimes I’d read a few paragraphs and think “I should probably spend the next decade trying to get that right” and sometimes I’d read 10 pages and think “that’s a long way of saying something that everyone already knows.”

Thanks to Ken Bubp, our team is also digging into the book together, which I hope will make us better.

Already, there are a couple of areas where the book made me want to get much better – a few of those detailed below.

The point of a work conversations is to find the truth

The point of social conversation is not  to find the truth: it’s to build bonds, have fun, and play status games.

These social habits tend to bleed into work conversations and it takes a lot of deliberate effort to create an environment where truth seeking is the paramount goal of a conversation.

Specifically, truth finding conversations require the implementation of two difficult to reconcile mindsets: not holding tight too tight to one’s own opinions + being aggressive enough to pressure test another person’s opinions.

So team members have to constantly toggle between:

you’re a smart person who shares my values so I need to figure out why you think I’m wrong because I might be wrong

and

I think you’re wrong because of X and I will not back down from this opinion for the sake of being agreeable

This is tough to do.

Normalize and embrace emotional and intellectual pain

In social relationships, we tend to avoid pain. Doing so is often bad for the long-term health of social relationships, and it can be devastating in professional settings.

Too often in work:

  1. It is considered harmful to cause someone else intellectual or emotional pain.
  2. The first reaction of feeling pain is often to attack others and build self-promoting narratives.

Both of these are very hard to unwind.

It’s very difficult to create a cultural where it is ok to cause someone the right kind of pain.

And it’s very hard to have the individual fortitude to build an internal monologue that goes like this: “the fact that I’m feeling a lot of pain means this is a prime opportunity for growth.”

With fluff and abuse at the polar ends of the cultural spectrum, getting this right is hard.

Leaders are defenders of the standard 

The standard for thinking and performance starts at the top, and the leadership of an organization must constantly live the standard and hold all others to the standard (this is also the principle of the great book The Score Takes Care of Itself).

It takes a lot of mental stamina to hold the bar: in 1-1 check-ins, in team meetings, in external conversations with partners…

…to never nod your head unless you mean it is flat out exhausting… to always ask the next probing question is annoying…. to say it’s not good enough yet is tiring…

But if you don’t do it you, your team, your organization will slide into mediocrity.

Getting the balance right between people, processes, and technology

Leaders have a finite amount of energy and capital to allocate across people, processes, and technology.

Different organizations, industries, and time periods require different allocations – as well as different types of expenditures.

Dalio goes deep in how he managed these as he grew his hedge fund, early on relying more on technology than most of his peers. While some his efforts are generalizable, many of them are not.

But the discipline by which he came to his approach is generalizable, and it’s something I want to keep working on as our team, technology, and industry evolve.

Application

After our team conversation, we’ll have to figure out how to apply these lessons. In some cases, it might simply be through consciously applying these principles in our conversations; in some cases, we’ll build processes; and perhaps down the road we’ll build out some technology.

SaveSave

Liberia is relinquishing. Is it working? 1st year results are in.

Screen Shot 2017-10-02 at 8.34.53 PM

Over the past few years, Liberia has embarked on an ambitious project to partner with non-governmental school networks.

Even more daring: Liberia’s political leadership is parterning with foreign school operators, some of which are for-profit.

Imagine for a second if, after Katrina, New Orleans political leaders had decided to partner with school operators from Singapore, Finland, and Shanghai.

In Liberia, numerous short-term and long-term risks abound – as do extremely high-potential upsides.

Partnership schools achieved .18 SD gains in one year 

Whatever one thinks of Liberia’s strategy, kudos to them for partnering with school networks in a manner that allowed for randomized control trials. Because schools were randomly selected for partnership, we can get a better understanding on whether or not the providers are delving a better education.

In aggregate, the first year effects were large: students in partnership schools scored 0.18 standard deviations higher in English and 0.18 standard deviations higher in mathematics than students in regular public schools. The authors note: “while starting from a very low level by international standards, this is the equivalent of 0.56 additional years of schooling for English and 0.66 additional years of schooling for math.”

Also, teachers are showing up more often: “teachers in partnership schools were 20 percentage points more likely to be in school during a random spot check (from a base of 40% in control schools).”

And teachers are teaching: “…16 percentage points more likely to be engaged in instruction during class time (from a base of 32% in control schools).”

Results varied by provider

The highest performing operators delivered ~.3 effects!: the Youth Movement for Collective Action (YMCA), Rising Academies, Bridge International Academies, and Street Child.

Second tier (very solid .15 effects): BRAC and More than Me.

Third tier (no effect): Omega and Stella Maris.

Screen Shot 2017-10-02 at 9.13.20 PM

Students receiving 2X learning time

This was incredible to me: “students in partnership schools spent twice as much time learning each week, when taking into account reduced absenteeism, increased time-on-task, and longer school days.”

Costs running a bit high (but to be expected in start-up)

The authors note that operators were spending more per-pupil than traditional schools; while this is a warning sign, I don’t read much into it now, as start-up efforts generally run higher and than smooth out. The exact same pattern happened in New Orleans.

What is the impact on traditional educators and schools?

As in the United States, non-governmental school operator growth impacts the traditional system, which has both programmatic and political consequences.

In an odd twist, the government contracts limited the class sizes of networks, which forced some operators to turn away students who then had to find other schools.

Operators also fired existing teachers, which presumably benefited children but caused adult hardship and risks political blowback.

I predict that these issues will only increase in salience. They require solutions that are both programmatic (government regulation of student equity issues) and political (ensuring that adult incumbents don’t derail positive efforts).

Teacher supply issues may get worse

The researchers note that to the extent that operators were able to recruit better teachers – and that the supply of teachers does not change – operators will be unable to scale and achieve the same effects.

In New Orleans we faced the same issue: we failed to grow high-quality teacher pipelines at the same pace we grew operators, and this caused operator growing pains midway through their scaling plans.

I hope Liberia gets ahead of this.

Is it worth it?

The perennial (and reasonable) question asked in such efforts is always: is it worth it? Is the disruption to families and educators worth the gain?

This question was asked a lot in New Orleans. I (as with the majority of New Orleanians) believe that it was worth it in New Orleans.

But I do think the Liberia case is more complicated, as it involves issues of national institutions and sovereignty.

There are numerous risks to outsourcing school operations to international organizations.

What if you end up in a conflict with the home nation(s) of large operators? What if these operators inculcate undesirable foreign values to your culture? What if the outsourcing of your educational operations slows down the overall maturation of your civil society?

These are hard questions.

My guess is that it is worth it, in that the gains of having a much better educated populace are worth the trade-offs of relying on foreign operators.

But I am not an expert in international development and I have not studied the issue enough to have strong opinions.

All that being said, all involved deserve our praise: the government is trying hard to serve their citizens, the school networks are serving students in extremely difficult situations, and the students themselves are getting smarter.

Here’s hoping the positive results continue.

If you want to be a superintendent build a school district

I cross paths with many people who want to become a superintendent of a large city school district.

Most of these people feel that this is the ultimate leadership position when it comes to serving children in need.

When I ask why, they say: “that’s where the kids are.”

I usually say: “this is not an immutable condition.”

Charter leaders are building some of the largest school districts in the country 

The most scaled high-performing charter network, KIPP, serves nearly a 100,000 students.

Right now, KIPP is around the 40th biggest school district in the country.

I bet within a decade it will be in the top 10 biggest school districts in the country.

A few other CMOs are on track to serve 100,000 students within the decade as well.

Within 10-15 years, a quarter of the top 25 biggest school districts in the country may be charter networks.

You can spend 15 years building an amazing school district or 3 years trying to fix a broken one 

KIPP is about 20 years old. Given all we know now (thanks in part to KIPP and other early CMOs), new charter founders should be able to hit the 100,000 student mark in less time.

With a bunch of hard and a bit of luck, the best entrepreneurs in the country should be able to replicate KIPP’s success and build 100,000 student CMOs in 10-15 years.

Compare this to being a superintendent: you inherit a struggling school district and have on average about 2-4 years to try and make it better before you are pushed out.

A few incredible superintendents succeed in making a dent, but most don’t.

As a charter founder, so much more of your potential for impact is in your control. And if you get results your board generally won’t fire you; rather, they’ll encourage you to serve more students.

There are about 10-15 million students attending public school and living in poverty

If the high-performing charter community could build 100 school districts that each served 100K students, we could provide nearly all students living in poverty with a great public education.

Are there a 100 people (or teams) in this country that can build a 100K school district? I don’t know.

Leading a major charter network is an incredibly difficult job. We need to do all we can be doing to make it as sustainable as possible.

The future of educational opportunity in this country might depend on it.

[thanks to James Cryan and Norman Atkins for inspiring this post]

CREDO’s school closure research validates portfolio and golden tickets

 

CREDO just came out with a study on school closures. Matt Barnum gives a good write up in Chalkbeat (and continues to far surpass NYT and WAPO in his analysis of complicated research).

Achievement increases when you close low performing schools and students transfer to better schools 

Overall, the research increased my belief in the idea that great schools should expand and failing schools should be closed or transformed (the basis of the portfolio model).

My only reservation with this study is that it defined low-performing schools by absolute performance rather than growth in achievement; however, student achievement still grew when students moved from lower to higher absolute performing schools, so perhaps many of the low absolute schools were low growth schools as well.

The research adds to the body of evidence that shows: if you…

(1) Close lower performing schools;

(2) Increase the number of high-performing schools;

(3) Ensure students from the closed schools get into the higher-performing schools; then

(4) Educational opportunity will increase.

See below from CREDO on the student achievement effect of a student transferring from a closed school to a superior school.

Screen Shot 2017-08-27 at 9.12.18 AM

Screen Shot 2017-08-27 at 9.12.29 AM

One other thing: this type of analysis doesn’t capture the positive effect of the failing school no longer existing; a city implementing these strategies should see additional gains from no new students ever having to attend the failing school.

“Golden ticket” policies can ensure students attending closed schools get into better schools

Cities that use unified enrollment systems can easily guarantee that students leaving closed schools have access to better schools by a “golden ticket” policy. This policy gives students exiting closed schools first access to any open seats in high-quality schools in the city.

While I find the policy name “golden ticket” to be crass, it’s a policy that can do wonders for educational equity.

Too often students in failing schools are shuffled from one underperforming school to another; a golden ticket policy can prevent this.

Should we support increased school closures in majority white communities?

Numerous commentators pointed out that schools with +80% minority students were more likely to be closed than schools that had lower minority enrollment.

Most took this as a sign of inequitable treatment toward minority communities.

However, a growing body of research indicates that school closure increases educational opportunity so long as student have access to better schools. And although I wish this wasn’t the case, my hunch is that majority white communities likely have a higher concentration of better schools than your average minority community: this means that school closures are likely to be even more effective in majority white communities.

My guess is a lack of closures in majority white communities is leading to reduced educational opportunity.

School closure is incredibly hard

While I believe that thoughtfully implemented school closure policies will benefit children, I know that closing schools is hard for students, families, educators, and politicians.

But sometimes doing the hard thing can help students.

And a growing body of evidence is pointing to the idea that a well implemented school closures is one of those hard things that can ultimately make things better for students and communities.

__

Lastly: thinking of everyone in Houston right now. Really hoping that friends, colleagues, and everyone else in city are ok.

SaveSave

SaveSave

Could the Earth ever become a Dark Forest?

In the trilogy The Three Body Problem, Liu Cixin builds his novels around the idea that the universe is a Dark Forest – i.e., when you’re moving through a dark forest and you hear the rustling of leaves, the optimal reaction is to shoot first.

More fully, the Dark Forest theory of the universe is built upon these first principles:

  1. The primary goal of each civilization is to survive.
  2. There are finite resources and space in the universe.
  3. Civilizations tend to expand.
  4. Civilizations tend to advance technologically.
  5. You have no way of truly knowing whether an alien species is peaceful or hostile.

So, if you detect an alien species – what do you do?

Under the Dark Forest theory, you kill them.

The reason you kill them is that even if they’re not hostile now, at some point they will want to survive, need more sources, and have advanced technology – which means they might just kill you.

____

I have no idea if the Dark Forest theory accurately describes the first principles of the universe.

But it made me think about something we might be able to understand with greater precision: could the Earth ever become a Dark Forest?

____

Right now, the Earth is not a Dark Forest largely because of nuclear deterrence.

North Korea might very well recognize that the existence of the United States will likely bring down their regime at some point, but they can’t act on this knowledge because we could respond to any nuclear attack with an attack that wipes them out.

Even for more robust nuclear powers, each side must live with the fact that a massive nuclear war could destroy all of humanity.

Culture also acts against the Earth becoming a Dark Forest. The scaling of large societies has been in part been sustained through cultural evolution: we now identify with nation and world instead of just kin, which, presumably, partially mitigates the 5th aforementioned principal (lack of trust).

But these conditions are not immutable: so it’s worth considering, how could one sided deterrence, assured mutual destruction, and trust…. end?

___

Unfortunately, it’s not hard to describe a scenario:

  1. There is a shortage of an important resource that is necessary to a nation’s survival, which makes securing that resource more important to a nation’s survival than the benefits of trade.
  2. This shortage, as well as the already significant cultural differences between existing rival nations (such as USA and China), erode trust.
  3. Technology advances in a manner that allows a nation that launches a first strike to kills all other humans, not allow for a return strike, and preserve the attacking nation.

How about this: there’s a water shortage that fuels nationalism, that leads to rising animosity between populous nations, and then one of them develops a synthetic virus that instantly kills all humans that haven’t received the vaccine – a vaccine that the attacking nation released in their own water supply the week before launching the attack.

___

I’m not in expert in these issues, so maybe I’ve gotten much wrong.

But if the universe can become a Dark Forest, the Earth probably can too.

If this is true, we’ll need a deterrence system for whatever set of weapons come after nuclear warheads.

But what are the odds that for every new weapon we develop we’ll also near simultaneously have an equally strong deterrence system?

They don’t seem high.

___

Please do let me know where my logic is off.