Category Archives: Accountability

Rational compassion is a competitive advantage

bloom2

Paul Bloom recently wrote a book called Against Empathy.

The thesis of the book is: rational compassion > empathy.

In other words: empathy (caring how someone feels at the moment) is poor guide for moral decision making when compared to rational compassion (which is more utilitarian in nature).

The difference is easiest to see when it comes to parenting: an overly empathetic parent might respond to a child’s failure by giving the child a cookie (thereby immediately decreasing the child’s suffering), while a parent utilizing rational compassion might help the child process her emotions (thereby reducing the probability of future instances of suffering).

While the idea is rather intuitive, we’re so hardwired for empathy that practicing rational compassion, especially at work, is very difficult.

Because it’s so hard to practice, and because most people are not good at it, the consistent use of rational compassion can be a competitive advantage for doing good in both the for-profit and non-profit sector.

List of Areas Where Rational Compassion > Empathy at the Work Place

Executing strategies that cause short-term harm for long-term gain: Tough decisions (such as school closures) cause short-term pain to others but can provide significant long-term outcomes. Being guided by rational compassion can help you get through this pain.

Pivoting and cannibalizing: Similarly, at times an organization needs to destroy existing program lines and harm existing beneficiaries of their work in order to pivot to a more productive model which will eventually add move value to more people (think Netflix going from mailbox to streaming). Empathy for existing employees and customers can blind one from the rationally compassionate act of eventually serving more people better.

Performance feedback: Rational compassion will lead you to give very direct and practical feedback so a colleague can improve her performance and achieve her and the organization’s goals. Having empathy for underperformance will lead to the avoidance of direct conversations, which in the short term causes more pain.

Firing people: Too much empathy for an individual who needs to be let go can cause immense harm to the people you are trying to serve. Especially in philanthropic work, firing a relatively privileged person in order to better serve people in extreme need is the rationally compassionate thing to do.

Accepting flaws of ambitious people: Sometimes ambitious people have a lot of flaws, which can lead you to empathize with all the people they are negatively impacting. However, these flawed people can also change the world for the better. Analyzing their actions through a rational compassion lens will help you understand if it’s worth supporting or partnering with people who are flawed but who can help the world become amazingly better. It will also help you avoid working deeply with nice people who are not effective.

The Risk of Rational Compassion 

One of the hardest parts of rational compassion is that it often involves overriding the legitimate short-term needs of others.

In other words: you’re saying you know what’s better for someone than she does.

While this is less of a tension in managerial situations (it’s your job to make feedback, coaching, firing decisions) and for-profit work (the customer will ultimately hold you accountable), in philanthropy (where it’s your job to help others) this can be a deadly sin.

It’s a blurry line between rational compassion and technocratic hubris.

There’s no easy way around this, though research and accountability can help.

In education, test scores, attainment, and parent demand can provide medium term feedback loops to provide a check on incorrect rational compassionate assumptions.

But while there are risks with rational compassion, most of society is so tilted toward empathy (especially in the education sector!) that an increase in the practice of rational compassion would be a welcome turn.

 

What if unified enrollment platforms were 10x better?

An emerging group of cities – including Washington D.C., Newark, Camden, New Orleans, and Denver – have adopted unified enrollment systems. With these systems, families can enroll in schools across the city via an online application system.

This is a huge step forward. For too long, parents have not had enough information or access to the public schools in their cities.

However, the new enrollment systems are still in their infancy. The best version of these systems could radically improve public education. Unfortunately, we’re very far from this endgame.

I. Early Wins: Access, Equity, and Ranking

Access: With the best open enrollment systems, families who can’t afford a house in a fancy neighborhood can now finally transparently apply to a school in a more wealthy neighborhood.

As a result of increase in access, a recent study in Washington D.C. found that the new enrollment regime would likely reduce segregation over time:

Screen Shot 2017-03-17 at 9.23.22 AM

Equity: In most cities, traditional and charter schools play a ton of games with enrollment. Traditional magnet schools use opaque entry requirements. Bad apple charters don’t take in kids with special needs. There is no equity.

With online enrollment platforms, these problems go away, as schools are no longer in control of their enrollment.

Quite simply: the algorithm is fairer than the enrollment clerk.

Ranking: These new enrollment systems also allow parents to rank their top schools. This is extremely important.

First, a family’s high desire to enroll their child in a school can now  be translated into an increased chance that they actually get into this school.

Previously, high desire meant little unless you were connected, wealthy, or dogged.

Second, ranking allows  parents to publicly signal to government which schools are most and least in demand (which will ideally affect opening, expansion, and closure decisions). It also signals to school operators what attributes make a school in high demand.

By analyzing ranking preferences, researchers in New Orleans were able to correlate school characteristics with parent preference:

Screen Shot 2017-03-17 at 9.44.39 AMRanking transforms family desire into actionable information.

II. Unified Enrollment Systems are Mediocre Platforms

In preparation for writing this blog, I spend an hour on unified enrollment system websites. It was not a great experience.

Here is the school finder homepage from Washington D.C. – I couldn’t even find a way to filter schools by academic performance!

Screen Shot 2017-03-17 at 9.50.14 AM

Here is the search function for Newark’s enrollment system – you have to download a pdf!

Screen Shot 2017-03-17 at 9.55.36 AM

By comparison, here’s the search page from Zillow:

Screen Shot 2017-03-17 at 9.53.10 AM.png

On Zillow, you can easily search for homes based on the most useful search criteria. Yes, Zillow is surely better capitalized than your average enrollment system, but even with modest funds a city should be able to do better than a downloadable pdf.

III. Moving From Equity and Ranking to Matching and Prediction

More sophisticated uniform enrollment could offer two extraordinary improvements: they could better match families with schools, and they could better predict how any given student would do at a school.

Matching: Right now families mostly use enrollment systems for ranking: they know the schools they want and they use enrollment systems to express this desire.

What is not really happening (as far as I can tell) is sophisticated algorithms actually helping families match with schools.

For example, on Camden’s enrollment site (where you can thankfully filter by academic performance!), I found three schools that all met the “on track” performance criteria, and pulled up the comparison page:

Screen Shot 2017-03-17 at 10.11.14 AM

This does very little to help me choose between these schools. My preference variables are limited to very broad categories such as “art classes” and “after school programs.”

After playing around on these websites, I get very little indication that that the platform knows me or the schools very well. Compare this to Netflix, Amazon, or dating websites (previous life) – platforms where I feel like the algorithms know me better than myself.

Unified enrollment systems need to more deeply understand children and schools in order to make better matches.

Prediction: Right now, government accountability systems are a basket case of poor design (generally don’t weight growth enough), brutal politics (what politician wants to tell a bunch of communities they only have “F” schools?), and awful transparency (good luck trying to navigate your average state department of education website).

Most importantly, government accountability systems evaluate schools rather than make predictions.

As a parent, it’s one thing to tell you that a school is a “C+” – it’s another thing to give you a prediction of what will happen to your child if she attends the school.

With current date, we could probably gather basic information on your child’s age, gender, current academic performance, personality type, etc.,  and make a reasonably accurate prediction that if she attends school X she will have a Y% chance of graduating from high schools and a Z% chance of earning a post-secondary degree.

Good enrollment systems, over time, should become better and better predictive agents, and, perhaps, can end up augmenting (displacing?) government accountability systems.

IV. Root Causes and Potential Solutions 

I don’t yet have strong beliefs about the root causes of why these enrollment products aren’t getting better faster. But here’s some guesses:

Non-profits > government operated: Most of the enrollment systems are run by governments, which are not good at running tech products and have bad incentives around giving parents accurate information about schools. Non-profits would likely be better operationally and have better incentives, and avoid the privacy concerns associated with for-profits.

Lack of scale: Matching and predication can better with bigger data sets, and if all these systems are structured as isolated city based data silos, the algorithms will be dumber than they should be.

Weak Customer Demand -> Bad Economics: SchoolMint, from what I understand, is the most successful player in the market. For reasons I don’t underhand, this company has not developed a better product. Perhaps it’s because their government customers don’t actually want it. Or perhaps the economics don’t work (which might suggest philanthropy is needed).

If the above is true, a national non-profit should be backed to scale to enough size to create smart algorithms, and it should be financially structured in a manner that gets it out of the perverse incentives of being beholden to government or individual schools rather than families.

A philanthropic foundation with a great tech backbone could be well situated to support this endeavor.

V. Expectations

Better matching and prediction would probably not make the average student’s education experience 10x better, just as dating websites don’t inevitably lead to great marriages.

But I do think better matching and predication could increase the probabilities that millions of families could find a better fit for their children.

At scale, that’s a better world.

A weak SIGnal: flawed research means we don’t know if SIG worked

I have a piece in Education Next about the study that came out the $7 billion federal school turnaround program.

Everyone is saying that the study proves SIG didn’t work.

I disagree.

My main argument is here:

In detailing these results, the authors note:

“The smallest impacts our benchmark approach could detect ranged from 0.19 to 0.22 standard deviations for test score outcomes, from 0.15 to 0.26 standard deviations for high school graduation, and from 0.27 to 0.39 standard deviations for college enrollment.”

Now, look back up at urban charter effects and you’ll see the three year results in math are about at the floor of what the SIG study could detect, and the results in reading are much lower than what the SIG study could detect (the SIG study also tracked children for 3 years).

So even if SIG achieved the same effects as urban charter schools the study may not have been able to detect these effects. 

It seems pretty unfair for charter (or voucher) champions to call SIG a failure when SIG might have very well achieved near the same results as urban charter schools.

My conclusion:

Until I see results that show that SIG worked, I won’t change my prior belief that SIG funds would have been better spent on high-quality charter growth.

Moreover, neither the existing research base nor theory warranted a $7 billion spend on district turnarounds, so even had the intervention worked I still would consider it a lucky outcome on an ill-advised bet.

But I also won’t claim that SIG failed.

Due to poor research design, we simply don’t know if that’s true.

The study authors, reporters, and commentators should walk back their strong claims on SIG’s failures.

At the same time, we should all keep advocating for government investment amounts to be in line with the existing evidence base.

If we have no reason to believe something will work, we should not spend $7 billion.

Too often, moonshots garner more status then they deserve.

Read the the whole piece here.

What Happens When What Works for Children Doesn’t Feel Good?

Closing schools does not feel good: it’s painful for families, educators, and politicians.

But closing schools, and opening new better schools, can dramatically help low-income children.

Sometimes, the best thing that can happen to a child is for her school to close.

Closing Schools Led to a +.3 SD Gain for Elementary Students in NOLA 

In New Orleans, Tulane researchers found that closing schools and creating new better schools led to very significant achievement gains:

screen-shot-2016-10-19-at-4-33-16-pm

Elementary students who attended a failing school started .1 SD behind their matched peers – two years later, these same students were .2SD ahead of their matched peers.

This +.3 SD impact is higher than the impacts of most educational interventions, and it equates to closing about 33% of the black-white student achievement gap.

You Don’t Have to Harm Existing Children for the Sake of Future Children

What’s incredible about these results is that the students whose schools were closed increased their student achievement.

Before seeing this data, my guess would have been that closing schools slightly harms existing students but is much better for future students who get to attend a better school without going through the disruption of closure.

But the NOLA data indicates that it’s possible to help both existing and future students, which should increase your belief in the benefits of school closure.

You Should Not Close Failing Schools and Send Children to Other Failing Schools 

In Baton Rouge, school closure did not lead to positive effects. This seems to be because these students enrolled into other failing schools after their original school was closed.

 

Screen Shot 2016-10-19 at 4.45.08 PM.png

For school closure to work, a city needs to either have open spaces in existing higher-performing schools or be opening new high-quality schools.

There are Good and Bad Ways to Close Schools

School closures are hard, but they can be done with respect. Families deserve to know why their schools are being closed; families should get support (and preference in unified enrollment systems) in finding a better school; and political leaders should ensure that empty school buildings are put to good use for the community.

Unfortunately, in many cities, political officials do not close schools thoughtfully. Instead of being honest with families about the poor performance of the school, they let failing schools linger for year until enrollment dwindles and the school folds academically and financially.

It is Difficult to Scale Something that Causes Political Pain

It is unclear to me whether or not deliberate school closure will scale. Reforms that cause political pain tend not to do well over time.

However, opening new great schools need not be politically painful, which bodes well for continued charter growth.

Of course, continued charter growth can lead to the closure of failing schools – and this is exactly why charter moratoriums have some political support.

Charter moratoriums have the potential to reduce pain for adults even as they inflict pain on children.

Can New Orleans Continue to Close Schools? Should It?

Over the past few years, academic performance has stagnated in New Orleans:

screen-shot-2016-10-21-at-8-20-25-am

During this time, there has also been a reduction in school closure activity.

So here is an interesting question: has the stagnation in performance been caused because New Orleans ate up most of the low-hanging fruit of closing schools, or has the stagnation in performance been caused because New Orleans has slowed down on closing failing schools?

At this point, I’m not familiar enough with the data to have strong opinions.

But I do worry that New Orleans, especially as it moves toward more local control, may stop using one of the strategies that has proven to dramatically improve the achievement of its students.

Every Mentor Should Become Your Peer

I. Rigorous Thinking and Effectiveness 

The most effective people that I’ve worked with, or engaged with online, are extremely rigorous thinkers.

This, in some sense, is satisfying.

To the extent that the connection between effectiveness and rigorous thinking is causation rather than correlation, then increasing the rigor of one’s thinking can help lead to greater effectiveness.

II. Implicit and Explicit Instruction 

Of the extremely rigorous thinkers I’ve interacted with, only some of them are good at explaining why they think the way they do.

All of them, on the other hand, have been good at telling me when my thinking has not been rigorous enough.

Given the above, I’ve spent a lot of time trying to take their implicit and explicit instruction in order to improve myself.

One incredibly positive aspect of the internet is that it opens up thousands of avenues to learn how other people think.

III. Mentorship 

Another way to say all this: the value of a mentor (either in person or online) is that she should significantly increase the rigor of your thinking.

When you communicate with her, you should run as many cycles of feedback as you can –  pitching your ideas, getting her critiques, and learning.

But, at some point, this can lead to laziness: you might rely on your mentor to do the thinking for you rather than doing it yourself.

One way to test your potential laziness is to continually take stock of the delta between the rigor of your thinking and the rigor of your mentor’s thinking. If you are not closing the gap between the rigor of your thinking and your mentor’s thinking, then you are, in some sense, exploiting your mentor.

Ultimately, every mentor should become your peer. This should be the goal of all involved.

IV. My Journey in Unrigorous Thinking 

For whatever it’s worth, my biggest obstacles to rigorous thinking have come from the following:

1. Yearning for Silver Bullets: Very few quick fixes exist for hard problems. In part, I think I was drawn to law school because of the idea that passing the right law could fix things quickly. As it happens, this is very rarely true. Hard problems tend to have decade long answers.

2. Yearning to Have My Problems Solved: Related but slightly different: when confronted with thorny day-to-day problems, I’ll sometimes gravitate toward the easiest or first solution rather than taking the time to rigorously analyze the issue. This is nothing more than intellectual laziness but it is difficult to control, especially when you’re moving fast and making a lot of decisions each day.

3. Ignoring Politics: I’ve also succumbed to spending a lot of time on ideas or programs that never had a chance of being political viable.

4. Ignoring Executing at Scale: Even ideas that might be politically viable may not be possible to scale operationally. Especially earlier in my career, I deeply underestimated execution as a limiting variable.

5. Wanting to be Agreeable: The desire to please or get along with others sometimes trumps my effort to push to get to the right answer.

Those are the big ones: I could have avoided many of the biggest mistakes of my career had I been more rigorous at avoiding these pitfalls.

I suspect I’m not the only person in my line of work to make these types of mistakes.

V. Thinking More Rigorously Hurts Until It Becomes a Habit 

For all of the above issues, there has either been a professional failure or extremely direct piece of feedback that has provided a wakeup call that I need to be thinking more rigorously.

Even after knowing I have a problem with the way I think, many times the only way I’ve been able to make progress  has been by being exposed to leaders who are adept at avoiding these pitfalls.

This often all rather painful.

For me, it literally hurts to think more rigorously in an area where I have not previously been a rigorous thinker.

In some instances it feels like being a child who knows his parents are right but doesn’t want to admit it even though it is in his best interest to (1) admit it and (2) incorporate his parent’s way of thinking into his world view.

Eventually, it gets less painful – and then it becomes a habit.

And then it’s on to the next one.

VI. Accountability is an Accelerant 

One last point: putting yourself into situations with high accountability for outcomes is one of the best ways to increase the rigor of your thinking.

If you are not accountable for outcomes, then you will be tempted to avoid the pain that comes with thinking more rigorously.

If you are accountable for outcomes, then the pain of potential failure helps offset the pain of thinking more rigorously.

The Great Charter School Clean Up?

 

Florida, Arizona, and Texas are known for having large charter school markets with large variation in quality.

Taking the first letter from each state name, let’s call these the FAT states.

All told, charter sectors in FAT states serve about 750,000 students (AZ = 180,000,FL = 280,000, TX = 280,000) – or about 25% of all charter students in the country.

Results in the FAT states have been mixed.

Here is what CREDO found in 2015 when they studied Texas charter school data:

Screen Shot 2016-02-06 at 10.13.34 AM

Even in urban areas, where charters usually perform the best, Texas charters lag compared to their traditional peers.

Previous CREDO studies in Arizona and Florida have found negative to mediocre results; however, more recent studies, especially those focusing on attainment, have found more positive results.

But, in terms of matched test results, the FAT states tend to poorly when compared to the charter sectors of Louisiana, Colorado, and Massachusetts.

___

The FAT states split the reform community.

For many of those whose posts show up on Jay Green’s blog, the FAT states are exactly what we need: high levels of entrepreneurship, disruption, and parent choice.

For many of those whose posts show up on CRPE’s blog, the FAT states have serious shortcomings: they represent the triumph of free market mania over the pragmatic restraints of quality control.

___

But something interesting is happening in the FAT states.

They are closing a lot of charter schools.

Last year, Florida closed 35 charter schools; Arizona closed 30 charters schools; and Texas closed 62 charter schools.

In Florida, regulators closed ~5% of all charter schools in a single year.

In Arizona, regulators closed ~6% of all charter schools in a single year. 

In Texas, regulators closed ~8% of all charter schools in a single year.

These rates are higher than the national charter school closure rates of ~4%.

In the case of Texas, their closure rate was double the national average.

___

We will learn much from the behavior of the FAT states (the next great charter paper must be lurking in this data).

Here are questions for which I would love to know the answer:

  1. How do the schools that are being closed compare to district schools peers in terms of academic growth, post-secondary attainment, earnings, and parent and student satisfaction?
  2. Over a long-period (25 years?) is it better for a state to let a thousand flowers bloom and then clean up the sector or to put on tight quality controls at the outset and then  allow for measured replication? Or somewhere in-between?
  3. How does the size of a charter sector affect its political support in the state legislature?
  4. How does the quality of a charter sector affect its political support in the state legislature?
  5. How does support in the state legislature affect quality control measures?

I’m sure there is more to be mined from the behavior of the FAT states.

Brown University vs. Science

The Annenberg Institute for School Reform at Brown University just released a report on state turnaround districts.

Brown University 

The report states:

In short, the Recovery School District, which was marketed (and continues to be lauded) as ushering in a miraculous transformation in New Orleans, has not kept its promise to some of the country’s most disadvantaged students.

The report cites another report, from Stanford University, and makes the following claims about equity and accountability:

The SCOPE [Stanford center] review also found that school quality and accountability are impeded by the lack of a strong central system (within the RSD) to support instructional improvement or maintain safeguards to ensure equity and access to reasonable quality of education.

Science 

Here is what Doug Harris, who actually studied student achievement in New Orleans, wrote:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Here’s a graph that captures these gains:

Screen Shot 2015-08-04 at 7.03.28 AM

Equity 

As for equity, I think this has been NOLA’s greatest innovation. I wrote a report on it. Here’s a highlight: despite serving a very at-risk student population, New Orleans has a lower expulsion rate than the state.

Screen Shot 2015-12-21 at 3.46.37 PM

Accountability 

As for accountability, it’s hard to think of a city that has been more serious about ensuring students don’t attend failing schools. In 2004, 60% of New Orleans students attended a school that was in the bottom 10% of the state. Now 13% do.

Screen Shot 2015-12-21 at 3.53.44 PM

People Who Live in New Orleans 

Lastly, here’s what New Orleanians think about the reforms:

Reasonable people can debate whether or not other states will see the same results.

But to say that the RSD has not kept its promise to the country’s most disadvantaged students is not supported by science.

Of course, there is still an incredibly long way to go in New Orleans. ACT scores, for example, are at an all-time high at around 19, but this still falls short of college and career ready.

Also, the New Orleans reforms were messy. While the academic results are undeniable, it’s been ten years of tense and difficult work, with many mistakes made along the way.

But if you don’t want things to be messy, you’re in the wrong line of work. The issues or race, class, and poverty are insanely complicated. If you work in the sector and haven’t changed your mind about a major issue, then you’re probably not thinking deeply enough.

All that being said… the student achievement gains are real. Children are better off.

And students across the country would be much better off if other cities achieved results similar to those in New Orleans.

Hopefully this will occur.