Category Archives: Decision Making

The problem with “risk / mitigation” sections in business plans

I see a lot of business plans and our team writes a lot of strategy memos.

I’ve always been a little frustrated with “risk / mitigation” sections of these documents, including some I’ve written myself.

“Risk / mitigation” sections feel like an exercise in confirmation bias. They almost always affirm the correctness of the business plan’s thesis.

The logic path of a risk / mitigation section is: “here’s our idea, there are some risks, but here’s why we’re going to do our idea anyway.

This is a modestly useful exercise. It’s worth understanding the risks of a plan and preparing for them. But this type of thinking does not force you to analyze the “crux” of your plan – i.e, what are the major trade-offs you’re making, and under what future conditions could you be proven wrong.

I prefer to see both of these lines of thinking in a business plan: first, a clear identification of the trade-offs in any major decision, and, second, a statement of what might make you change your mind about your plan’s thesis.

Any very hard decision will include difficult trade-offs. And any hard decision might also result in you being wrong. Identifying these ahead of time will help you be mentally prepared to make difficult shifts down the road.

It will also decrease the probability that you drink your own kool aid.  There is a humbling effect in writing down potential future conditions that will indicate that you are near failing or have already failed.

I hope that one day every business plan includes “trade-off” and “we will know we were wrong if” sections.

Doubling down when you’ve been wrong

From 2012 to 2016, three reforms took place in Memphis: the Gates Foundation teacher evaluation reform, the Achievement School District (ASD) charter school turnaround effort, and the district run izone turnaround effort.

In private conversations, I predicted that neither the teacher evaluation nor izone efforts would work. I thought the ASD charter effort work would work. The Arnold Foundation funded research to help us understand if our beliefs about how to help children were correct.

Five years later, the results paint a different picture than my predictions: the teacher evaluation work did not improve achievement. The ASD charters have yet to deliver results. And the izone schools have done the best.

I was wrong about the izone and the ASD.

So what to do now?

The short answer: expand non-profit governance to protect the gains of izone schools, and continue to grow the best non-profit public charters and replace the worst.

The longer answer is below.

Responding to Being Wrong

There are three ways to respond to being wrong: ignore that you were wrong, change your behavior, or admit that you were wrong but double down.

Ignoring that you’re wrong is never good.

Changing your behavior is often the most reasonable response.

Doubling down comes with substantive and reputation risk, but it can be the right thing to do.

Memphis Results 

The teacher evaluation reforms went roughly as I expected, so I won’t discuss that too much here, other than saying that my predictions were based on the belief that execution would hamper implementation in big urban districts, which seems to be exactly what happened.

Here are the results of the izone and ASD charter efforts:

 

 

This slideshow requires JavaScript.

The izone had a positive .2 stand deviation effect over five years. This is fairly good impact and it’s great to see these schools helping students.

The ASD schools had no impact.

What I Got Wrong

I made a few mistakes.

First, I underestimated the school district’s strategic and executional competency. In creating the izone, the district successfully recruited a lot of their best educators to their izone schools, and then appointed a talented leader, Sharon Griffith, to oversee these schools. This strategy (concentrating talent) and execution (selecting a great leader) worked in ways I did not expect.

Second, I overestimated the abilities of charter networks to deliver results in the first few years of the turnaround efforts. My assumption was that expert charter leaders would be able to deliver better results pretty quickly, even though they were going into very difficult situations and were often coming from out of town. Unfortunately, a few operators really struggled in the early years, with one of them, Yes College Prep, choosing to not even open up a school (this was a big hit as YES is considered to be one of the best networks in the country).

What I Would Have Done Differently 

I should have realized that great educators in the traditional system could achieve good results if given autonomy in a competitive environment. I have changed my beliefs on this issue based of the work of the izone, the work in Lawrence, Massachusetts, and my lived experience of seeing great teachers and leaders in all types of schools.

Rather than predict the izone wouldn’t work, I should have predicted that the izone might work, but advocated for non-profit governance over the zone so that it could sustain its gains. This remains my worry with the izone: that it will not be able to sustain its impacts over the next 5-10 years due to its governance structure, which makes it vulnerable to political shifts.

I also worry that it can’t really scale. Putting the best educators in the worst school is more of a one time intervention rather than a strategy that can grow over time, and it has some negative effects on the schools that lose their great educators.

Second, I should have realized that early stage charter operators (either newly started or new to Memphis) would struggle to scale through full school turnarounds in a difficult political environment. In hindsight, I should have advocated for more one grade at a time roll outs of newer operators. While this would have been more disruptive to families at the outset, it would have been better for kids over the long run.

What to Do Now

If the above diagnosis is true, it doesn’t really call for a dramatic change in strategy.

Rather, it calls for trying to get non-profit governance over the best district schools, growing the charters that are working, replacing those that aren’t, and starting new non-profit district and charter schools under more favorable conditions.

Getting non-profit governance to the best district educators would (I think) require a change in legislation.

Growing the best charter schools and closing the worst  can be done under current policy.

Evidence to Support Doubling Down

Researchers have found evidence of charter school sectors improving over time. This paper on charters in Texas found significant sector improvements, which led the researchers to write:

The findings suggest the value of taking a longer-term perspective when evaluating the impact of a major educational reform such as the introduction of charter schools, especially when the success of the reform ostensibly depends on parental decisions and market forces.

Additionally, previous research on Memphis charters found strong positive effects:

Screen Shot 2018-06-23 at 8.03.44 AM

These results mirror the results of many other urban charter sectors, which generally find positive effects.

So there is a research based case for predicting that the Memphis charters will improve over the next 5-10 years. We have already seen this with some operators, such as Aspire.

In Sum 

The research has increased my belief that great district school educators, with autonomy and support (and often pressure on local districts from the state to give this autonomy) can really help struggling students.

I still believe that these gains risk backsliding due to political shifts, so I support giving these schools non-profit governance so they can build enduring organizations (both Denver and Indianapolis have mechanisms to give district schools non-profit status).

I also still believe that growing the best charters and closing the worst is one of the best hopes for improving student achievement in Memphis.

So, in many ways, I’m choosing to double down despite being wrong, with the caveat of calling for non-profit governance for more types of public schools.

While this is uncomfortable, I don’t know a better way forward. I think doubling down will help children.

I understand that many of those who I disagree with say “we just need more time.” And I’m making a version of this argument here. So I’ve tried to lay out all my assumptions as clearly as possible, both for the sake of transparency, as well as with the hope that these assumptions can be corrected if they are wrong.

If you support neighborhood schools you also (unintentionally) support segregated schools

A reminder: for the foreseeable future, supporting neighborhood schools means de facto supporting segregated schools.

The reason is obvious: neighborhoods in our country are highly segregated.

I think our country would be better if our neighborhoods weren’t segregated, but I don’t think that’s going to change anytime soon.

So if every kid goes to their neighborhood school, we will have segregated schools.

There are a couple ways out of this.

We could restructure enrollment and bussing rules to avoid segregation, but this would mean that a lot families would have to attend schools outside of their neighborhoods.

And  I don’t think it’s the rich families that are going to send their children to poorer neighborhoods.

So we’ll need to be bussing in low income students into rich neighborhoods. This might be the right thing to do, but that means a lot of low income students won’t be attending their neighborhood schools.

It also means we’ll likely have a lot of white flight, which, while unfortunate, is neither good for integration nor a city’s tax base.

The other option is to create all choice systems and allow schools to preference students in a way that increases socioeconomic integration (this could be done through a unified enrollment algorithm).

For this strategy to be successful, schools will have to proactively create enrollment rules that increased integration, and families will have to proactively choose schools with this mission.

This will obviously be a slower process than forced integration via non-choice bussing systems.

But I think it will be much more durable.

Ultimately, you can’t force people to integrate our current version of segregated schools if they don’t want to. They will either move or kick out the superintendent who forces it. As a country, we rightfully changed the laws that forced segregation, but we’re still left with the fact that many people don’t really want integration, at least not if it involves any bit of giving up of privilege.

So, no, you can’t force integration. But you can give educators the opportunity to say that their school will prioritize integration. And you can make it easier for families to choose these schools.

The road to school integration is not through neighborhood schools. And it’s not through forced enrollment patterns.

The road to school integration is through people actually wanting it, and for government to create open systems that allow these desires to be actualized.

If you do nothing, people will attend their segregated neighborhood schools.

If you force it, they will flee.

If you build it, they might come.

The Current Brutal Reality of Education Reform and Wage Growth

edu wealth

Going from 16K to 18K in Annual Wages 

Last week, I did a post on Fryer and Debbie’s excellent new study on the Texas charter sector.

I emailed the authors about my hypothesis that the growth of high-quality charters – even if they aren’t that much better than average traditional schools – could still be of great value if these new charters displaced chronically failing schools.

Roland was kind enough to respond but pushed that even if my hypothesis is true, the story still might be a depressing one.

His point: the numbers from his study indicate that even if we replaced all these failing schools with high-performing charters, we’re still only talking about ~1-2K in extra earnings per year for these students.

Given that many of these students end up lower income brackets, this might mean going from 16K to 18K a year in annual salary. Hardly game changing in terms of life outcomes – and surely not a ticket to the middle class.

Confronting this Potential Reality

When a study tells you what you don’t want to hear, the first reaction is often to not deal with it (in some ways I did this in my previous post).

So everyone in education reform needs to deal with this potential reality: there is some possibility that the best that education reform has to offer can only, on average, move a student from 16K to 18K a year.

Of course, this is only one study of one state. We don’t yet know if these numbers will hold under different contexts, methodologies, or timeframes.

But, at the very least, your belief that a great school can radically increase wages should be a little lower after reading this study.

Other Considerations

I’m still mulling this over, but in conversations with Roland and folks I work with, certain ideas bubbled up:

The data doesn’t capture recent improvements: A lot of the best charters have only really started focusing on college and career over the past 5 years or so. As such, the students who received the full suite of redesigned high schools, counseling, and career support aren’t represented in this study. To the extent you believe the best charters are problem solving machines, you might believe this to be true.

The work is generational: Perhaps reformed schools can only, on average, push students who would have been in deep poverty to achieve average poverty / lower middle-class status. And perhaps their children, who will grow up in better educated environments, will the be ones  to more fully make it into the middle class. But this story could be unwound through raised expectations: if we told kids they were going to make it to the middle class, and they don’t, how will they react?

Colleges are the bottleneck: Perhaps these real gains in learning are being wasted by ineffective two year and four year colleges – and that without higher education reform we won’t be able to translate K-12 gains into wage increases.

Society is tough: Just because you’re better educated doesn’t mean you can overcome racism, lack of social capital, and an over-reliance on signaling.

More interventions are needed: Great schools can’t solve everything; interventions that work on family poverty, health, and parenting are needed for schools to really move kids as far as they need to be moved.

The schools aren’t really that good: A bunch of teaching to the test just jacks up crystallized knowledge but doesn’t really give kids the human capital qualities they need to succeed in the workforce.

What Do you Do in the Face of Ambiguity?

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

The Answer is 6.7 Miles. What is the Question?

Screen Shot 2016-06-15 at 8.01.52 PM

The question is: how far, on average, would a family send their child to attend a school that is in the highest category of the state accountability system compared to a school in the lowest category of the state accountability system?

This is from a recent report on the DC public school system. The analysis, while useful, isn’t perfect in that it only includes families who utilized the enrollment system, but it does add to the emerging literature on the revealed preferences of families that participate in transparent enrollment systems.

 

___

Here’s another answer: it increases racial integration.

The question is: does DC’s unified enrollment system increase or decrease racial segregation?

Screen Shot 2016-06-15 at 8.06.39 PM

Shockingly enough, assigning families to neighborhood schools that are zoned by property values is not a great way to decrease segregation.

 

___

Answer: Unclear.

Question: Do parents care about a school’s academic growth (as opposed to absolute test scores)?

Screen Shot 2016-06-15 at 8.20.46 PM

Interesting but not shocking. Parents probably care a lot about peers and status.

Also interesting, this seems more true of low-income families:

Screen Shot 2016-06-15 at 8.25.26 PM

This raises an interesting question for policy makers: given that growth more accurately measures a school’s impact, should they design grading systems that prioritize growth (as DC’s charter framework does) even though low-income parents might care more about absolute scores?

Or perhaps not – maybe low-income families aren’t considering the growth based performance framework because the government is hiding this information:

Screen Shot 2016-06-15 at 8.27.05 PM

___

One last answer: Families who aren’t assigned to a school in the lowest performance category, as well as the politicians and superintendents who seek their favor.

The question: who loves neighborhood schools?

It remains shocking to me that public leaders in cities such as Oakland are vehemently opposed to unified enrollment on the grounds that such systems will undermine public education.

The only thing a unified enrollment system undermines is the privilege of those who benefit from institutional racism and widespread income inequality.

 

 

Is No Excuses or Personalized Learning the Low Hanging Fruit of School Improvement?

On average, I visit a school every other week or so. For the most part, these schools are equal to or better performing than the median urban district school.

During these visits, one question I usually mull over is this: if I was leading the school, what would I focus on to drive the next phase of improvement?

Often times, what the school leader is focusing on and what I would focus on are at odds.

I don’t have extremely high confidence in my analysis, so consider the below speculative.

____

Here are the things I most often here school leaders saying they need to improve on: personalization, student ownership, and critical thinking. Tactically speaking, this often leads them to experiment with new models of instruction and technology.

All good things.

But I’m often thinking that the school really needs to get better at: instructional delivery, higher ratios of student intellectual engagement, and more effective use of small group instruction.

Most school leaders seem to believe that they have the basics down and need to go from good to great.

I tend to think that most schools are mediocre at the basics of things such as cold calling, wait time, efficient time on task, and tutoring – and the other hall marks of the no excuses model.

___

So here’s some questions on my mind:

  1. Should the median charter school be focusing on getting better at the basics of the no excuses model or experimenting with deeper innovation?
  2. If it’s true that the median charter school is still mediocre at the no excuses basics, what should we take from this? That high fidelity to the no excuses basics is operationally hard to scale for either intellectual, emotional, or human capital reasons? That many leaders don’t think the no excuses basics work?
  3. Is there a progression of improvement (i.e., you need to get the basics right before you work on deeper innovation) – or does shifting to more innovative models allow you to bypass the no excuses basics and still get academic gains?

___

My hunch is this: only the top tier charter organizations and the very best entrepreneurs should be working deeply on the margin of innovation.

Most charter schools should be working on the margin of better adoption of the tenets of the no excuses model.

Once new models are hammered out and refined – and get better results than the no excuses model – then the median charter school should begin adopting these new models.

But not before that.

In sum, I think better fidelity to the no excuses model is the low-hanging fruit of school improvement.

Maybe I’m wrong? Maybe I’m very wrong?

The Challenge of Separating Emotional and Intellectual Agreeableness

There is a decent amount of research showing that agreeableness (as measured by the five factor personality test) is not always associated with strong professional outcomes.

Specifically, agreeableness can reduce results orientation and create opportunities to be taken advantage of by colleagues who better use power to achieve their desired ends.

That being said, agreeableness need not be all bad: to the extent that it helps cultivate large, loose networks, agreeableness is likely of use to leaders in attracting talent and coalition members, especially in the non-profit sector.

___

Many times, I’ll be in a conversation with a colleague, grantee, or potential grantee and there will be a small war going on my head: part of me wants to nod my head, smile, and ask probing but pleasant questions – while another party of me wants to dig in very hard on everything that might be wrong about what we’re discussing.

I have a strong desire to be both emotionally agreeable and intellectually disagreeable.

Which begs the question: is it possible to be emotionally agreeable while being intellectually disagreeable?

___

I’m not sure. But here’s some things I try to do:

  • Utilize processes that create a safe space for intellectual aggression (i.e., assigning someone to be the devil’s advocate in a meeting).
  • Using hedging phrases such as “I might have this wrong, but….” that soften the blows of intellectual aggression.
  • Trying to separate my empathy for a person with my disagreement with her ideas – so that my intellectual disagreeableness does not bleed into full blown personal animosity.

If you have any other tools, let me know.

I struggle to get the balance right.

Sometimes I feel like I’m too agreeable, and sometimes I feel like I’m too intellectually aggressive.