Category Archives: Hard Things

How much I gave to charity this year -> and to which cause -> and why my giving might be mistaken

2 Replies

Every year I write a post about how much I give to charity. I consider this an act of positive virtue signaling. If we’re going to compete on something, competing on how much we give to charity is the right kind of competition.

This year I’m slightly altering my reporting. Instead of reporting this year’s giving, I’m reporting a five year charitable giving percentage. I consider this a more honest reporting, as it smoothes out year to year fluctuations.

Over the past five years, I’ve given away 8.5% of my total five year pre-tax earnings.

How does this compare to your giving? I’d love to hear about how much you give and what you give to in the comments.

What I Give To: Expanding Bed Net Access to Reduce Malaria 

Most of my giving goes to the Against Malaria Foundation. They are recommended highly by Givewell.

I donate to AMF because there is good evidence that bed nets save lives and because my marginal contribution increases the number of people who have bed nets. Despite the massive success of bed nets, there is still an on-going need.

Researchers studied the decline of cases of malaria in Africa between 2000 and 2015. They found that the single most important contributor to the decline were insecticide-treated bed nets.

Bed nets were responsible for the aversion of 68% of the 663 million averted cases in Africa between 2000 and 2015. These are 451 million averted cases. Given that children make up 72% of malaria fatalities, this is a truly remarkable impact for families.

Screen Shot 2018-11-22 at 12.57.25 PM

Additionally, researchers estimate that the malaria “penalty” to GDP ranges from 0.41% of GDP in Ghana to 8.9% of GDP in Chad, all of which could be regained following elimination of malaria.

Not only does my gift potentially saves lives, it is also positively impacts economic productivity.

All together, Givewell estimates (very roughly), that every $4,000 spent on bed nets saves a life.

If this is true, and I keep up my giving, I will be able to save a lot of lives.

As a new father, I can barely comprehend what it would be like to lose our child. I hope my giving will over time help hundreds of families avoid the pain and suffering caused by one of life’s worst tragedies.

When I think about whether or not it’s worth it to give, I think about our daughter.

Two Reasons (Out of Many) I Might Be Wrong

It is hard to help other people. I’ve tried to minimize this risk by giving in an area with  lots of evidence, low operational complexity, and clear health benefits.

But the fact is bed nets are never going to get people out of extreme poverty.

The only way for people living in extreme poverty to get out of extreme poverty is through rapid economic growth. Bed nets will not cause rapid economic growth.

The problem is that I have no idea what will cause massive economic growth in Africa.

But here are somethings I have considered funding:

Economic Research: Lant Pritchett makes the case that economic research allows us to learn truths that help countries escape poverty; i.e., the research on the benefits of trade, property rights, and other liberal economic principles have led to many countries adopting these policies, which has led to massive increases in wealth. Perhaps the same could be said of the research on domestic industry subsidization that forces subsidized companies to export competitively (some say this is a key driver for the Asian tigers). I could fund this research in the United States, or work with others to set-up research programs in local universities. A few friends and I could probably cobble together enough money to fund a full-time professor at a prestigious African university to work on these issues.

Technological Innovation: Technological progress is a primary cause of wealth creation. People living in Africa have much longer lifespans today because of technological innovations invented elsewhere. While my giving alone probably isn’t enough to impact technological research or venture capital investing, I’ve wondered about trying to get a group of 50 people or so and invest alongside established funds that are dedicated to technological innovation in globally important areas, such as energy. It’s plausible that in the case of investing, I could even get my money back and do a lot of good.

How I Feel About Giving

For the most part, giving makes me feel good. It feels morally correct to reduce my consumption so I can save the lives of children living in poverty.

But it also stings a bit. If you put together all the various taxes I pay, my tax burden is somewhere between 40-50% (such is life in California!). When you add my charitable contributions to this, that’s nearly 60% of my income out the door.

I also sometimes worry about my family. I live a very comfortable life and don’t want for anything. But life is unpredictable and this could change. If I or a loved one were in a severe accident, it’s quite plausible that I could run through my savings in under a decade. Giving to charity now reduces my ability to withstand big shocks later. Ultimately, I view the ability to withstand big shocks as a privilege that shouldn’t trump my duty to help others now, but it’s still something I worry about.

So there it is.

I give away 8.5% of my pre-tax income and I allocate much of it to malaria reduction. I hope this helps others in need.

 

 

Should states use test score based accountability systems? If so, how? If not, why?

3 Replies

Over the past decade, I’ve deepened my belief in the power of letting educators form non-profits to run public schools. Both experience (walking into amazing public schools) and research (a track record of reading and math gains) have shown me that non-profits are an incredibly valuable tool in making public education better.

I’ve also deepened my belief in unified enrollment systems. They can give families a lot of information about public schools and make enrolling in public schools much easier.

I do not have deep confidence in my views on accountability. I often find myself moving up and down the spectrum of: no accountability (just let parents choose), to accountability-lite (require testing, share this information, but don’t intervene), to accountability heavy (require testing, give schools letter grades, intervene in lowest performing schools).

I think reasonable arguments can be made for all three approaches.

Recent NWEA Research

NWEA just published a new report using a national data set from the tests they license to schools. Many schools we work with use these tests. I’m not expert enough in statistics to evaluate the reliability of their findings, but the report raised some important issues.

Absolute test scores are highly correlated with poverty. The chart below shows that test scores rise as income increases. This is not new information.

Screen Shot 2018-10-04 at 7.17.05 AM

Student growth is not tightly correlated with poverty. Unlike absolute achievement, individual student growth does not rise significantly with income. Many high poverty achieve growth that mirrors those of their wealthier peers.

Screen Shot 2018-10-04 at 7.19.26 AM.png

Schools with similar levels of poverty perform very differently on growth. The red line in the chart below represents how schools with high poverty perform on academic growth. It is a fairly wide curve. Many schools achieve low growth, while others achieve very high growth. To the extent you believe that growth is a pretty good measure of school performance (the researchers do), this performance spread might increase a policymaker’s willingness to intervene in low-performing schools and expand high-performing schools.

Screen Shot 2018-10-04 at 7.23.15 AM.png

Focusing on absolute test scores will cause you to misidentify many, many schools. The graph below is tricky to read, but it’s very important. The red line represents all schools that are in the bottom 5% for absolute test scores. And it shows that 77% of these schools (the bottom 5% on absolute) are close to the average or better on growth. In other words, if you just closed the bottom 5% of schools based on absolute achievement, nearly 80% of the schools you’d close probably would be mistakenly closed (given their growth scores). This is pretty damning evidence against those who want to focus mostly on absolute achievement in accountability measures.

Screen Shot 2018-10-04 at 7.27.08 AM

When Does a Good Policy Idea Become Indefensible Because of Bad Practice?

Over the past few years, most states reworked their accountability systems during the reauthorization of No Child Left Behind.

Unfortunately, this report found that only 18 states weighted growth for at least 50% of the total accountability score, with another 23 states weighting growth at least at 33%.

On one hand, this is an improvement over old accountability systems. On the other hand, this means a lot of states are unfairly rating high poverty schools that have decent growth but low absolute scores.

I think a fair critique of test based accountability is that it’s a reasonable idea that has very little hope of being reasonably implemented.

My Own Thoughts

Again, I do believe deeply in letting non-profit organizations operate public schools. And I do believe deeply in enrollment systems that make it easier for families to find a great school for their children.

I’m uncertain about accountability, but here’s what I think I’d do if I were superintendent of a school district:

  1. Calculate a letter grade score for growth and a letter grade score for absolute achievement score.
  2. Publish the higher of these grades as the letter grade that appears most prominently on the online enrollment system. I would also include the lower letter grade, as well as a bunch of information about school programs and curriculum, on the school’s online profile.
  3. Allow for government intervention in schools that are in the bottom 5-10% for both growth and absolute (you need to perform bad on both).

This type of accountability system gives parent’s good information, avoids the political war of giving low letter grades to schools with high absolute scores, and avoids the error of intervening in schools that have low absolute scores and higher growth scores.

It does give an accountability pass to schools with high absolute scores and low growth, but I view this ok in that it’s both politically useful and it does reflect the notion that parents really want to get into these schools.

It also still uses test scores as the primary way to evaluate schools. This sits uneasy with me, as I think schooling is about much more than tests, but I haven’t seen any other way to measure schools that feels more reliable. I hope this changes.

I’m not very confident that this is the best system, but I think it’s the best of a bunch of options that all have reasonable drawbacks.

Another hard question would be what to do if local politics did not allow for the creation of a system like this. At some point, if the drum beat for absolute scores was too much, I’d probably walk away from accountability as a superintendent.

But I’m not sure. If you scan this blog’s history, I’m sure you can find me saying conflicting things about accountability. I’m conflicted about it. But the above reflects my current thinking of what makes for a good accountability system.

Lastly, if you want to hear a good version of the argument against test based accountability, see here.

The problem with “risk / mitigation” sections in business plans

Leave a reply

I see a lot of business plans and our team writes a lot of strategy memos.

I’ve always been a little frustrated with “risk / mitigation” sections of these documents, including some I’ve written myself.

“Risk / mitigation” sections feel like an exercise in confirmation bias. They almost always affirm the correctness of the business plan’s thesis.

The logic path of a risk / mitigation section is: “here’s our idea, there are some risks, but here’s why we’re going to do our idea anyway.”

This is a modestly useful exercise. It’s worth understanding the risks of a plan and preparing for them. But this type of thinking does not force you to analyze the “crux” of your plan – i.e, what are the major trade-offs you’re making, and under what future conditions could you be proven wrong.

I prefer to see both of these lines of thinking in a business plan: first, a clear identification of the trade-offs in any major decision, and, second, a statement of what might make you change your mind about your plan’s thesis.

Any very hard decision will include difficult trade-offs. And any hard decision might also result in you being wrong. Identifying these ahead of time will help you be mentally prepared to make difficult shifts down the road.

It will also decrease the probability that you drink your own kool aid.  There is a humbling effect in writing down potential future conditions that will indicate that you are near failing or have already failed.

I hope that one day every business plan includes “trade-off” and “we will know we were wrong if” sections.

Doubling down when you’ve been wrong

2 Replies

From 2012 to 2016, three reforms took place in Memphis: the Gates Foundation teacher evaluation reform, the Achievement School District (ASD) charter school turnaround effort, and the district run izone turnaround effort.

In private conversations, I predicted that neither the teacher evaluation nor izone efforts would work. I thought the ASD charter effort work would work. The Arnold Foundation funded research to help us understand if our beliefs about how to help children were correct.

Five years later, the results paint a different picture than my predictions: the teacher evaluation work did not improve achievement. The ASD charters have yet to deliver results. And the izone schools have done the best.

I was wrong about the izone and the ASD.

So what to do now?

The short answer: expand non-profit governance to protect the gains of izone schools, and continue to grow the best non-profit public charters and replace the worst.

The longer answer is below.

Responding to Being Wrong

There are three ways to respond to being wrong: ignore that you were wrong, change your behavior, or admit that you were wrong but double down.

Ignoring that you’re wrong is never good.

Changing your behavior is often the most reasonable response.

Doubling down comes with substantive and reputation risk, but it can be the right thing to do.

Memphis Results 

The teacher evaluation reforms went roughly as I expected, so I won’t discuss that too much here, other than saying that my predictions were based on the belief that execution would hamper implementation in big urban districts, which seems to be exactly what happened.

Here are the results of the izone and ASD charter efforts:

 

 

This slideshow requires JavaScript.

The izone had a positive .2 stand deviation effect over five years. This is fairly good impact and it’s great to see these schools helping students.

The ASD schools had no impact.

What I Got Wrong

I made a few mistakes.

First, I underestimated the school district’s strategic and executional competency. In creating the izone, the district successfully recruited a lot of their best educators to their izone schools, and then appointed a talented leader, Sharon Griffith, to oversee these schools. This strategy (concentrating talent) and execution (selecting a great leader) worked in ways I did not expect.

Second, I overestimated the abilities of charter networks to deliver results in the first few years of the turnaround efforts. My assumption was that expert charter leaders would be able to deliver better results pretty quickly, even though they were going into very difficult situations and were often coming from out of town. Unfortunately, a few operators really struggled in the early years, with one of them, Yes College Prep, choosing to not even open up a school (this was a big hit as YES is considered to be one of the best networks in the country).

What I Would Have Done Differently 

I should have realized that great educators in the traditional system could achieve good results if given autonomy in a competitive environment. I have changed my beliefs on this issue based of the work of the izone, the work in Lawrence, Massachusetts, and my lived experience of seeing great teachers and leaders in all types of schools.

Rather than predict the izone wouldn’t work, I should have predicted that the izone might work, but advocated for non-profit governance over the zone so that it could sustain its gains. This remains my worry with the izone: that it will not be able to sustain its impacts over the next 5-10 years due to its governance structure, which makes it vulnerable to political shifts.

I also worry that it can’t really scale. Putting the best educators in the worst school is more of a one time intervention rather than a strategy that can grow over time, and it has some negative effects on the schools that lose their great educators.

Second, I should have realized that early stage charter operators (either newly started or new to Memphis) would struggle to scale through full school turnarounds in a difficult political environment. In hindsight, I should have advocated for more one grade at a time roll outs of newer operators. While this would have been more disruptive to families at the outset, it would have been better for kids over the long run.

What to Do Now

If the above diagnosis is true, it doesn’t really call for a dramatic change in strategy.

Rather, it calls for trying to get non-profit governance over the best district schools, growing the charters that are working, replacing those that aren’t, and starting new non-profit district and charter schools under more favorable conditions.

Getting non-profit governance to the best district educators would (I think) require a change in legislation.

Growing the best charter schools and closing the worst  can be done under current policy.

Evidence to Support Doubling Down

Researchers have found evidence of charter school sectors improving over time. This paper on charters in Texas found significant sector improvements, which led the researchers to write:

The findings suggest the value of taking a longer-term perspective when evaluating the impact of a major educational reform such as the introduction of charter schools, especially when the success of the reform ostensibly depends on parental decisions and market forces.

Additionally, previous research on Memphis charters found strong positive effects:

Screen Shot 2018-06-23 at 8.03.44 AM

These results mirror the results of many other urban charter sectors, which generally find positive effects.

So there is a research based case for predicting that the Memphis charters will improve over the next 5-10 years. We have already seen this with some operators, such as Aspire.

In Sum 

The research has increased my belief that great district school educators, with autonomy and support (and often pressure on local districts from the state to give this autonomy) can really help struggling students.

I still believe that these gains risk backsliding due to political shifts, so I support giving these schools non-profit governance so they can build enduring organizations (both Denver and Indianapolis have mechanisms to give district schools non-profit status).

I also still believe that growing the best charters and closing the worst is one of the best hopes for improving student achievement in Memphis.

So, in many ways, I’m choosing to double down despite being wrong, with the caveat of calling for non-profit governance for more types of public schools.

While this is uncomfortable, I don’t know a better way forward. I think doubling down will help children.

I understand that many of those who I disagree with say “we just need more time.” And I’m making a version of this argument here. So I’ve tried to lay out all my assumptions as clearly as possible, both for the sake of transparency, as well as with the hope that these assumptions can be corrected if they are wrong.

Did a federal grant to turnaround failing schools in New Orleans and Tennessee work?

Leave a reply

Back when I worked at New Schools for New Orleans, we applied for a $30m federal grant to turnaround failing schools in New Orleans and scale the model to Tennessee.

CREDO just came out with a research study on our efforts. Their findings, and my analysis, are below.

The New Schools Were Much Better than the Ones They Replaced 

Here’s what CREDO found when they compared the schools we created to the schools we replaced:

In New Orleans, we replaced schools (“closing schools”) that were at 26th percentile in the state with new schools (“CRM schools”) that performed at the ~33rd percentile in the state at the end of the study.

In Tennessee, schools went from the ~17th percentile to the ~23rd percentile by the end of the study.

To quote the CREDO report: “the CRM schools in both New Orleans and Tennessee showed significantly higher academic growth compared to the Closing schools they replaced.”

Translated into days of learning, these are large effects: “Closing school students experience 63 fewer days of learning in reading and 86 fewer days of learning in math when compared to students in non-CRM schools… students in CRM schools make comparable academic growth to non-CRM students.”

The New Schools Performed About the Same as Other Schools in the City

When CREDO compared the new schools to other existing schools (rather than the failing schools they replaced), they found no statistically significant effects:

Screen Shot 2018-02-15 at 12.07.57 PM.png

In other words, the new schools that replaced the failing schools performed no better or worse than other existing schools in the city.

On one hand, this is disappointing. Our most ambitious targets included having the new schools be amongst the highest performing schools in the city.

On the other hand, this is still a major improvement: the new schools replaced failing schools and ended up achieving at the same level of most other schools in the city.

Building a System that Keeps Getting Better 

Replacing failing schools with new schools is a process, not a one-time intervention.

Ideally, a subset of the schools you created will do really well, and then, overtime, these schools will continue to grow. The ones that don’t do well will not be supported to do additional turnarounds.

Over the long-haul, gradually increasing the number and scale of high-quality school operators is more important than the average effect of the first wave of replacements.

Here’s what CREDO found across the new schools when they compared them to existing schools:

Screen Shot 2018-02-15 at 12.11.36 PM.pngIn New Orleans, 50% of the new schools had positive effects in both Math and Reading. This is really positive: half of our turnaround schools in New Orleans achieved significantly better results than existing schools across the city.

In Tennessee, only one school had positive effects in both Math and Reading, though a few other schools had positive effects in only reading.

This makes me optimistic that the school operator base in New Orleans will continue to have the capacity to replace more failing schools over time.

The early results in Tennessee are a bit more worrying on the operator quality front, and the next few years will be extremely important in ensuring that a healthy operator base emerges.

Lastly: CREDO found that replacing failing schools with fresh start schools (that opened one grade at a time) had a higher success rate than whole school turnarounds. My takeaway here is that you need a mature operator base to do a lot of whole school turnarounds, and no city had enough capacity to really do whole school at scale. In hindsight, we should have done more fresh starts and less whole school turnarounds.

Was the Effort a Success?

At the outset of the project, I remember debating with our research partners at CREDO about how to set-up the evaluation.

I argued that we should ultimately be judged on whether or not the new schools we created were better than the failing schools we replaced.

I didn’t think we should be primarily judged on whether or not the new schools were better than other existing schools that weren’t failing.

Yes, we did include language in the grant application that had goals of schools performing much better than existing schools. And as we executed the project we tried to pick school operators that we thought could deliver top tier results. Our highest aspirations weren’t met. This is disappointing, but it does not mean the project was a failure.

Rather, I consider the project to be a positive step forward in improving public education in these cities.

Making Things Better

The result of the project strikes at the heart of what’s so difficult about education reform: our aspirations for our most at-risk children are incredibly high, but making progress in creating better educational opportunities is very difficult.

In roughly a five year period, we replaced failing schools with new schools that were on average 7 percentile points higher in state performance, which translates to an extra 60-90 days of learning per year.

If the process of opening and replacement continues, what is a modest success right now may eventually become a great success.

I hope that this occurs and that New Orleans continues on its impressive track record of increasing student achievement. As a reminder, the federal grant was just one piece of an overall effort that has radically reduced failing schools in the city:

Screen Shot 2018-02-20 at 5.47.30 PM

SaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSave

How should CEOs handle what society demands?

Leave a reply

As I get older, my illusion of control weakens.

This is generally a good thing: the more I acknowledge that I can’t change much of the world, the more focus I have on the few areas where I might actually do some good.

For me, this means less time reading the news and more time on work and personal community.

But even within areas of some control, such as work, there are a lot of limits.

One of those limits is how, as a leader, to deal with what society demands.

Society Makes Demands 

I recently had dinner with No Excuses charter founder, and he was describing how his school’s attempts to implement restorative justice have been a train wreck. After two years of cultural decline, the school is now just getting back to having a culture of high expectations that helps children learn.

I don’t know whether his was because of poor implementation or actual serious flaws in the restorative justice model itself. But I do know that there was a lot of pressure for the school to adopt the restorative justice model.

So let’s put aside the idea of whether or not restorative justice is a good cultural model for a school  – rather, let’s consider the larger question: what should a CEO do when society demands something they disagree with?

In this case, liberal society (from which most No Excuses teachers come from) is increasingly demanding a more progressive school culture, with a lot of young white teachers reacting negatively to having to manage (what feels like to them) overly authoritarian cultures for black children.

For the sake of argument, let’s say you’re the CEO of a charter organization and you disagree with what society is demanding. What should do?

Bend or Stay Firm?

What if you think that restorative justice will lead to a decline in culture that hurts children’s lives? What do you do?

This is a very difficult question.

If you stay firm, you might lose the talent pipelines that had made you so successful in the first place. Or you might lose a communications battle that significantly reduces family demand for you school.

In other words, even if you think what society is demanding is wrong, it might still make sense to bend to society’s demands and just try to mitigate the negative impacts.

In this case, you’re basically trying to ride it out. Society’s demands constantly change – and you just have to hope that this moment in time will pass – and you can course correct in the future.

On the other hand, it might be the case that if you stay firm you will prove society wrong: when everyone else’s results plummet, you’ll be knocking it out of the park.

There are never any easy answers when you disagree with what society demands.

Some Advice 

 

Too often, CEOs make the mistake that society, having made its demand, is now willing to hear the CEO’s response.

By the time society has made its demands, it’s usually not in a contemplative mood.

So here’s some advice:

Consider bending: The best course of action is often to bend to society in a way that keeps your organization afloat and lets you live to fight another day.

Have a values conversation: If you decide this is a place where you really need to make a stand, the default position should be to always engage society in a values conversation, not a strategy conversation. Society doesn’t make demands about operations and strategies, it makes demands about values and tribal affiliations. If you’re going to try and convince society you’re right, you need to win on values.

Make sure your team is with you: If you’re making a values argument, you’ll get clobbered if a bunch of people internal to your organization say that they disagree with your values – or provide reporters with a bunch of examples of how you’ve violated the values you’re now professing to hold.

To the extent I’ve had to push back on society’s demands (keep neighborhood schools, don’t close schools), I’ve always tried to do it in a values based way. And I’ve always tried to surround myself with people who share these values. And, at times, I’ve bended: such as supporting enrollment systems that give a partial neighborhood preference to families… and respecting the demands of people who hold different values than I do.

Does anyone know why Chicago children are getting smarter?

3 Replies

If you just read the headlines, you might suspect that Chicago’s public schools are in a terrible tailspin. Part of this is the noise of big city politics. Part of this stems from cloud of violence that hangs over the city.

But Chicago has improved on academic test scores more than most other cities in the country. Rather than one of the worst, Chicago is one of the best.

A recent report by Sean Reardon and Rebecca Hinze-Pifer found that between 2009 and 2014:

“This [student achievement] growth rate [of Chicago] is higher than 96% of all districts in the US. Among the 100 largest districts in the country, the average growth rate from third to eighth grade is 0.95 grade equivalents per year; Chicago has the highest growth rate between third and eighth grade of any large district in the United States.”

The authors admit that they don’t know why this occurred.

I can’t prove why Chicago kids are getting smarter, but I have a hypothesis.

What’s Been Going on in Chicago Public Schools? 

One way to try and solve the mystery of why Chicago children are getting smarter is to look at the district’s previous major initiatives.

As this report details, between roughy 1990 and 2010 there were three overarching eras of reform in Chicago: the decentralization era, the the accountability era, and the do a lot of things era.

The authors are very careful to not attribute a causal relationship between reform eras and outcomes. The reforms were messy and not rolled out in an experimental manner – so fair enough.

But in this post I’ll try and make my best guess on what was causal and what was not.

The Decentralization Era

The decentralization era was best known for the creation of Local School Councils. This reform gave local councils real control over decisions about how schools were run. The councils were made up of school leadership, parents, and community members.

The councils always seemed like a terrible idea to me. It’s basically taking all we know about charter schools (good central offices, scalable instructional programs, governance matters) and doing the exact opposite!

Not surprisingly, research on the reforms found that the councils had some positive effects on advantage communities, but were least likely to improve schools in low-income communites. Communities with low social capital didn’t gain a lot from ad-hoc and poorly constructed local boards.

I’m very skeptical that the decentralization era and school councils were the root cause of later gains.

The Accountability Era

In 1995 Mayor Daly put in Paul Valls as the superintendent (I later worked with Paul when he was the superintendent of the RSD in Louisiana). Vallas, who did not have deep instructional expertise, used test driven accountability to try and make things better.

New tests, promotional standards, and interventions for failing schools were all put in place.

The reforms had better impacts for low-performing schools; the researchers noted:

“This was the only era to show large improvements in the lowest- achieving schools. However, the patterns in test scores in the lowest-performing schools suggest that some of the improvements resulted from instruction that was aligned specifically to the high stakes tests.”

This matches other research on accountability reforms: you tend to see gains in the lowest preforming schools, but the high stakes can cause narrowing of the curriculum.

 

The Do a Lot of Things Era

Arne Duncan came in after Vallas, and he instituted a lot of reforms.

Arne launched 100 new schools, implemented internal district instructional and curricular reforms, overhauled school leadership pipelines, and placed a deep focus on on-time high school progression.

Perhaps the biggest initiative of this era was the Renaissance 2010 project, which launched about a 100 new district, charter, and contract schools between 2005 and 2010.

Unfortunately, no one has conducted a full evaluation of the program. Someone should do this!

Two interim research reports came out around 2010. One study, which only included a few years of data from the early Renaissance cohorts, found that the new schools performed about the same as the existing district schools. The other study was inconclusive.

Not much help from the research community.

A lot of work was also done on school leadership. The Chicago Public Education Fund, in partnership with the district, invested heavily in school leader development, placing bets on both district based and non-profit providers.

The lastest research I could find on these programs found that “results indicate that one-year learning gains in elementary and high schools led by Fund-supported principals were not different than those in other similar schools.”

Another major reform, another mediocre result.

All told, researchers found that this era produced more gains in high school than elementary schools, but wrote: “while the effects of the dominant policies of Eras 1 and 2 are largely understood, much research remains to be done to understand both the positive and problematic effects of the policies in Era 3.”

Not super helpful, especially since this is the era that preceded the large gains in test scores that occurred after 2009.

What About the Charter Sector? 

CREDO published a report on Chicago charters that covered test scores from 2010 to 2012, which is right in the middle of the period where Chicago saw a lot of gains.

The study found +.01 effects in reading and +.03 effects in math. These effects amount to about a month or so of extra learning per year, maybe a bit less. Given Chicago’s relatively small charter market share, and the modest size of these positive effects, it’s unlikely that charters themselves accounted for the 2009-2014 gains.

A more recent study, which just looked at charter high school performance from 2010-2013, found much larger effects: +.2 effects on ACT related tests and much higher college enrollment rates.

These are large effects, but they are for high school only. The study lauding Chicago’s gains only covered grades 3-8.

So WTF Happened in Chicago to Make Kids Smarter?

To summarize: Chicago improved its test scores more than any other big city in the country, and researchers really don’t know why.

So why are Chicago kids getting smarter?

Here’s my guess: competition and accountability lifted all boats.

When you put accountability in place (the Vallas era) and then launch a 100 new schools (the Duncan era) you get a city where school leaders know there are consequences for failure and the best of the new schools begin to raise the bar for what’s possible.

This theory helps explain why the Renaissance schools and charter effects were a bit muted. In the studies on these reforms, researchers compared the new schools to existing schools. So if the existing schools were improving due to increased competition, you would not see large relative effects for the new schools.

I can’t prove that accountability and competition caused the results, but in many sectors accountability and competition make everyone better. It also fits stories we’ve seen elsewhere. In place like Denver and Washington D.C. increased competition led to all boats rising in the public school system.

If you have a better theory, let me know.

What Should Chicago Do Now? 

Here’s another tough question: if it was accountability and competition that caused Chicago’s gains, how should this impact Chicago’s future strategy?

Since 2002 (while the district was getting much better!) Chicago enrollment plummeted from 440,000 students to 370,000 students.

This means that there are lot of under-enrolled schools in the district and the city might have to go through another round of painful closures.

This also it means it’s harder to push the very reform (opening new schools) that might have driven Chicago’s previous gains in achievement.

So what should the city do?

Reasonable people can surely disagree, but I would continue to create new schools, albeit in a different fashion.

First, I’d open new schools in the areas where population is increasing. Chicago is made up of a lot of neighborhoods, and not all neighborhoods are losing children.

Second, I would do some replacement work. Instead of closing all the under-enrolled schools, I’d try and select some neighborhoods where there’s enough child density that you could imagine families coming back to the public schools if there were better options. I’d launch replacement schools in these neighborhoods.

There are clear drawbacks to this strategy. Politically, it’s hard to justify opening schools when you’re in the midst of closures. Programmatically, it’s hard sell to get the operators of new schools to open up in neighbors with shrinking enrollment.

But I think it’s the best thing for children.

Lastly, I might also try and launch some diverse by design schools.

In a city as diverse as Chicago, it’s sad that it’s schools are so segregated.

The Last Word

Chicago’s Chief Education Officer, Janice Jackson, recently gave her take on why things are better.

Her list: pre-k, better professional development, better curriculum, competition from private and charter schools, and clear accountability standards.

In her own words:

“I believe the level of transparency we have provided around what a quality school is has been transformational in this district.”

A thoughtful response from Chris Cerf

2 Replies

Chris Cerf, the current superintendent of Newark and the former state superintendent of New Jersey, posted a thoughtful response to my post on the Newark Harvard study.

A couple of thoughts.

First, I’ve had the opportunity to interact with Chris on and off over the past decade, and I have immense respect for his heart and mind. Kids in New Jersey are better off because of his leadership.

Second, I appreciated the tenor of Chris’ post. A primary reason I write this blog is so that  people in the education reform family can have public disagreements and learn from each other. Chris’ tone and use of data helped me get smarter on Newark.

While there is some cost to this approach (those who oppose our work can try to exploit our disagreements), ultimately, I think the gain of learning through public debate is well worth it.

Over the long haul, our success will have more to do on whether or not we continually delivered great educational opportunities for children than whether or not we win or lose the PR fire of the week. And putting our ideas out there for public testing is a good way to get smarter on how we deliver better opportunities for children.

As for the substance of Chris’ points, you should read the piece for yourself, but I found the below two graphs to be useful.

The first compares Newark’s overall performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s relative performance to similarly situated districts has improved greatly over the past seven years.

newark-graph1

The second compares Newark’s traditional school performance to similarly situated districts in New Jersey (DFG A -> the green line). Newark’s traditional school performance was fairly flat until 2014, but has grown rapidly since then.

newark-graph2

I’m not sure that Chris and I disagree on the data story.

Both of us, I think, would say that the early gains in Newark were driven by the open / close / shift strategy.

As for the improve the traditional sector strategy, Chris points to the last few years of growth to demonstrate that the reforms are starting to deliver for all schools – and that now that the foundation has been set, these gains will likely continue.

I emphasized the flat results for the first four years of the district improvement strategy and wondered whether the improve strategy was worth it – or whether putting more resources into the open / close / shift strategy would have laid an even better foundation for long-term gains.

It’s a great question to grapple with.

Lastly, our interpretations of the study go to an even more fundamental question of how we measure success: should a city’s reform efforts be evaluated on the cumulative gains it achieved during the transformation process, or should it be evaluated on the gains being delivered once a city is through a transformation process?

When we partner with a city, our team holds ourselves for cumulative gains over the initial 5-10 years of reform, but perhaps a more important metric is whether a city achieves a new and better equilibrium by the end of the reforms.

Chris has pushed me to think hard about this question, our team will be smarter for it.

Book Review: Ray Dalio’s Principles

Leave a reply

dalioo

How to read it

I just finished Ray Dalio’s and found the book to be both uneven and very useful.

Sometimes I’d read a few paragraphs and think “I should probably spend the next decade trying to get that right” and sometimes I’d read 10 pages and think “that’s a long way of saying something that everyone already knows.”

Thanks to Ken Bubp, our team is also digging into the book together, which I hope will make us better.

Already, there are a couple of areas where the book made me want to get much better – a few of those detailed below.

The point of a work conversations is to find the truth

The point of social conversation is not  to find the truth: it’s to build bonds, have fun, and play status games.

These social habits tend to bleed into work conversations and it takes a lot of deliberate effort to create an environment where truth seeking is the paramount goal of a conversation.

Specifically, truth finding conversations require the implementation of two difficult to reconcile mindsets: not holding tight too tight to one’s own opinions + being aggressive enough to pressure test another person’s opinions.

So team members have to constantly toggle between:

“you’re a smart person who shares my values so I need to figure out why you think I’m wrong because I might be wrong”

and

“I think you’re wrong because of X and I will not back down from this opinion for the sake of being agreeable”

This is tough to do.

Normalize and embrace emotional and intellectual pain

In social relationships, we tend to avoid pain. Doing so is often bad for the long-term health of social relationships, and it can be devastating in professional settings.

Too often in work:

  1. It is considered harmful to cause someone else intellectual or emotional pain.
  2. The first reaction of feeling pain is often to attack others and build self-promoting narratives.

Both of these are very hard to unwind.

It’s very difficult to create a cultural where it is ok to cause someone the right kind of pain.

And it’s very hard to have the individual fortitude to build an internal monologue that goes like this: “the fact that I’m feeling a lot of pain means this is a prime opportunity for growth.”

With fluff and abuse at the polar ends of the cultural spectrum, getting this right is hard.

Leaders are defenders of the standard 

The standard for thinking and performance starts at the top, and the leadership of an organization must constantly live the standard and hold all others to the standard (this is also the principle of the great book ).

It takes a lot of mental stamina to hold the bar: in 1-1 check-ins, in team meetings, in external conversations with partners…

…to never nod your head unless you mean it is flat out exhausting… to always ask the next probing question is annoying…. to say it’s not good enough yet is tiring…

But if you don’t do it you, your team, your organization will slide into mediocrity.

Getting the balance right between people, processes, and technology

Leaders have a finite amount of energy and capital to allocate across people, processes, and technology.

Different organizations, industries, and time periods require different allocations – as well as different types of expenditures.

Dalio goes deep in how he managed these as he grew his hedge fund, early on relying more on technology than most of his peers. While some his efforts are generalizable, many of them are not.

But the discipline by which he came to his approach is generalizable, and it’s something I want to keep working on as our team, technology, and industry evolve.

Application

After our team conversation, we’ll have to figure out how to apply these lessons. In some cases, it might simply be through consciously applying these principles in our conversations; in some cases, we’ll build processes; and perhaps down the road we’ll build out some technology.

SaveSave

Liberia is relinquishing. Is it working? 1st year results are in.

1 Reply

Screen Shot 2017-10-02 at 8.34.53 PM

Over the past few years, Liberia has embarked on an ambitious project to partner with non-governmental school networks.

Even more daring: Liberia’s political leadership is parterning with foreign school operators, some of which are for-profit.

Imagine for a second if, after Katrina, New Orleans political leaders had decided to partner with school operators from Singapore, Finland, and Shanghai.

In Liberia, numerous short-term and long-term risks abound – as do extremely high-potential upsides.

Partnership schools achieved .18 SD gains in one year 

Whatever one thinks of Liberia’s strategy, kudos to them for partnering with school networks in a manner that allowed for randomized control trials. Because schools were randomly selected for partnership, we can get a better understanding on whether or not the providers are delving a better education.

In aggregate, the first year effects were large: students in partnership schools scored 0.18 standard deviations higher in English and 0.18 standard deviations higher in mathematics than students in regular public schools. The authors note: “while starting from a very low level by international standards, this is the equivalent of 0.56 additional years of schooling for English and 0.66 additional years of schooling for math.”

Also, teachers are showing up more often: “teachers in partnership schools were 20 percentage points more likely to be in school during a random spot check (from a base of 40% in control schools).”

And teachers are teaching: “…16 percentage points more likely to be engaged in instruction during class time (from a base of 32% in control schools).”

Results varied by provider

The highest performing operators delivered ~.3 effects!: the Youth Movement for Collective Action (YMCA), Rising Academies, Bridge International Academies, and Street Child.

Second tier (very solid .15 effects): BRAC and More than Me.

Third tier (no effect): Omega and Stella Maris.

Screen Shot 2017-10-02 at 9.13.20 PM

Students receiving 2X learning time

This was incredible to me: “students in partnership schools spent twice as much time learning each week, when taking into account reduced absenteeism, increased time-on-task, and longer school days.”

Costs running a bit high (but to be expected in start-up)

The authors note that operators were spending more per-pupil than traditional schools; while this is a warning sign, I don’t read much into it now, as start-up efforts generally run higher and than smooth out. The exact same pattern happened in New Orleans.

What is the impact on traditional educators and schools?

As in the United States, non-governmental school operator growth impacts the traditional system, which has both programmatic and political consequences.

In an odd twist, the government contracts limited the class sizes of networks, which forced some operators to turn away students who then had to find other schools.

Operators also fired existing teachers, which presumably benefited children but caused adult hardship and risks political blowback.

I predict that these issues will only increase in salience. They require solutions that are both programmatic (government regulation of student equity issues) and political (ensuring that adult incumbents don’t derail positive efforts).

Teacher supply issues may get worse

The researchers note that to the extent that operators were able to recruit better teachers – and that the supply of teachers does not change – operators will be unable to scale and achieve the same effects.

In New Orleans we faced the same issue: we failed to grow high-quality teacher pipelines at the same pace we grew operators, and this caused operator growing pains midway through their scaling plans.

I hope Liberia gets ahead of this.

Is it worth it?

The perennial (and reasonable) question asked in such efforts is always: is it worth it? Is the disruption to families and educators worth the gain?

This question was asked a lot in New Orleans. I (as with the majority of New Orleanians) believe that it was worth it in New Orleans.

But I do think the Liberia case is more complicated, as it involves issues of national institutions and sovereignty.

There are numerous risks to outsourcing school operations to international organizations.

What if you end up in a conflict with the home nation(s) of large operators? What if these operators inculcate undesirable foreign values to your culture? What if the outsourcing of your educational operations slows down the overall maturation of your civil society?

These are hard questions.

My guess is that it is worth it, in that the gains of having a much better educated populace are worth the trade-offs of relying on foreign operators.

But I am not an expert in international development and I have not studied the issue enough to have strong opinions.

All that being said, all involved deserve our praise: the government is trying hard to serve their citizens, the school networks are serving students in extremely difficult situations, and the students themselves are getting smarter.

Here’s hoping the positive results continue.