Category Archives: Accountability

What Happens When What Works for Children Doesn’t Feel Good?

Closing schools does not feel good: it’s painful for families, educators, and politicians.

But closing schools, and opening new better schools, can dramatically help low-income children.

Sometimes, the best thing that can happen to a child is for her school to close.

Closing Schools Led to a +.3 SD Gain for Elementary Students in NOLA 

In New Orleans, Tulane researchers found that closing schools and creating new better schools led to very significant achievement gains:


Elementary students who attended a failing school started .1 SD behind their matched peers – two years later, these same students were .2SD ahead of their matched peers.

This +.3 SD impact is higher than the impacts of most educational interventions, and it equates to closing about 33% of the black-white student achievement gap.

You Don’t Have to Harm Existing Children for the Sake of Future Children

What’s incredible about these results is that the students whose schools were closed increased their student achievement.

Before seeing this data, my guess would have been that closing schools slightly harms existing students but is much better for future students who get to attend a better school without going through the disruption of closure.

But the NOLA data indicates that it’s possible to help both existing and future students, which should increase your belief in the benefits of school closure.

You Should Not Close Failing Schools and Send Children to Other Failing Schools 

In Baton Rouge, school closure did not lead to positive effects. This seems to be because these students enrolled into other failing schools after their original school was closed.


Screen Shot 2016-10-19 at 4.45.08 PM.png

For school closure to work, a city needs to either have open spaces in existing higher-performing schools or be opening new high-quality schools.

There are Good and Bad Ways to Close Schools

School closures are hard, but they can be done with respect. Families deserve to know why their schools are being closed; families should get support (and preference in unified enrollment systems) in finding a better school; and political leaders should ensure that empty school buildings are put to good use for the community.

Unfortunately, in many cities, political officials do not close schools thoughtfully. Instead of being honest with families about the poor performance of the school, they let failing schools linger for year until enrollment dwindles and the school folds academically and financially.

It is Difficult to Scale Something that Causes Political Pain

It is unclear to me whether or not deliberate school closure will scale. Reforms that cause political pain tend not to do well over time.

However, opening new great schools need not be politically painful, which bodes well for continued charter growth.

Of course, continued charter growth can lead to the closure of failing schools – and this is exactly why charter moratoriums have some political support.

Charter moratoriums have the potential to reduce pain for adults even as they inflict pain on children.

Can New Orleans Continue to Close Schools? Should It?

Over the past few years, academic performance has stagnated in New Orleans:


During this time, there has also been a reduction in school closure activity.

So here is an interesting question: has the stagnation in performance been caused because New Orleans ate up most of the low-hanging fruit of closing schools, or has the stagnation in performance been caused because New Orleans has slowed down on closing failing schools?

At this point, I’m not familiar enough with the data to have strong opinions.

But I do worry that New Orleans, especially as it moves toward more local control, may stop using one of the strategies that has proven to dramatically improve the achievement of its students.

Every Mentor Should Become Your Peer

I. Rigorous Thinking and Effectiveness 

The most effective people that I’ve worked with, or engaged with online, are extremely rigorous thinkers.

This, in some sense, is satisfying.

To the extent that the connection between effectiveness and rigorous thinking is causation rather than correlation, then increasing the rigor of one’s thinking can help lead to greater effectiveness.

II. Implicit and Explicit Instruction 

Of the extremely rigorous thinkers I’ve interacted with, only some of them are good at explaining why they think the way they do.

All of them, on the other hand, have been good at telling me when my thinking has not been rigorous enough.

Given the above, I’ve spent a lot of time trying to take their implicit and explicit instruction in order to improve myself.

One incredibly positive aspect of the internet is that it opens up thousands of avenues to learn how other people think.

III. Mentorship 

Another way to say all this: the value of a mentor (either in person or online) is that she should significantly increase the rigor of your thinking.

When you communicate with her, you should run as many cycles of feedback as you can –  pitching your ideas, getting her critiques, and learning.

But, at some point, this can lead to laziness: you might rely on your mentor to do the thinking for you rather than doing it yourself.

One way to test your potential laziness is to continually take stock of the delta between the rigor of your thinking and the rigor of your mentor’s thinking. If you are not closing the gap between the rigor of your thinking and your mentor’s thinking, then you are, in some sense, exploiting your mentor.

Ultimately, every mentor should become your peer. This should be the goal of all involved.

IV. My Journey in Unrigorous Thinking 

For whatever it’s worth, my biggest obstacles to rigorous thinking have come from the following:

1. Yearning for Silver Bullets: Very few quick fixes exist for hard problems. In part, I think I was drawn to law school because of the idea that passing the right law could fix things quickly. As it happens, this is very rarely true. Hard problems tend to have decade long answers.

2. Yearning to Have My Problems Solved: Related but slightly different: when confronted with thorny day-to-day problems, I’ll sometimes gravitate toward the easiest or first solution rather than taking the time to rigorously analyze the issue. This is nothing more than intellectual laziness but it is difficult to control, especially when you’re moving fast and making a lot of decisions each day.

3. Ignoring Politics: I’ve also succumbed to spending a lot of time on ideas or programs that never had a chance of being political viable.

4. Ignoring Executing at Scale: Even ideas that might be politically viable may not be possible to scale operationally. Especially earlier in my career, I deeply underestimated execution as a limiting variable.

5. Wanting to be Agreeable: The desire to please or get along with others sometimes trumps my effort to push to get to the right answer.

Those are the big ones: I could have avoided many of the biggest mistakes of my career had I been more rigorous at avoiding these pitfalls.

I suspect I’m not the only person in my line of work to make these types of mistakes.

V. Thinking More Rigorously Hurts Until It Becomes a Habit 

For all of the above issues, there has either been a professional failure or extremely direct piece of feedback that has provided a wakeup call that I need to be thinking more rigorously.

Even after knowing I have a problem with the way I think, many times the only way I’ve been able to make progress  has been by being exposed to leaders who are adept at avoiding these pitfalls.

This often all rather painful.

For me, it literally hurts to think more rigorously in an area where I have not previously been a rigorous thinker.

In some instances it feels like being a child who knows his parents are right but doesn’t want to admit it even though it is in his best interest to (1) admit it and (2) incorporate his parent’s way of thinking into his world view.

Eventually, it gets less painful – and then it becomes a habit.

And then it’s on to the next one.

VI. Accountability is an Accelerant 

One last point: putting yourself into situations with high accountability for outcomes is one of the best ways to increase the rigor of your thinking.

If you are not accountable for outcomes, then you will be tempted to avoid the pain that comes with thinking more rigorously.

If you are accountable for outcomes, then the pain of potential failure helps offset the pain of thinking more rigorously.

The Great Charter School Clean Up?


Florida, Arizona, and Texas are known for having large charter school markets with large variation in quality.

Taking the first letter from each state name, let’s call these the FAT states.

All told, charter sectors in FAT states serve about 750,000 students (AZ = 180,000,FL = 280,000, TX = 280,000) – or about 25% of all charter students in the country.

Results in the FAT states have been mixed.

Here is what CREDO found in 2015 when they studied Texas charter school data:

Screen Shot 2016-02-06 at 10.13.34 AM

Even in urban areas, where charters usually perform the best, Texas charters lag compared to their traditional peers.

Previous CREDO studies in Arizona and Florida have found negative to mediocre results; however, more recent studies, especially those focusing on attainment, have found more positive results.

But, in terms of matched test results, the FAT states tend to poorly when compared to the charter sectors of Louisiana, Colorado, and Massachusetts.


The FAT states split the reform community.

For many of those whose posts show up on Jay Green’s blog, the FAT states are exactly what we need: high levels of entrepreneurship, disruption, and parent choice.

For many of those whose posts show up on CRPE’s blog, the FAT states have serious shortcomings: they represent the triumph of free market mania over the pragmatic restraints of quality control.


But something interesting is happening in the FAT states.

They are closing a lot of charter schools.

Last year, Florida closed 35 charter schools; Arizona closed 30 charters schools; and Texas closed 62 charter schools.

In Florida, regulators closed ~5% of all charter schools in a single year.

In Arizona, regulators closed ~6% of all charter schools in a single year. 

In Texas, regulators closed ~8% of all charter schools in a single year.

These rates are higher than the national charter school closure rates of ~4%.

In the case of Texas, their closure rate was double the national average.


We will learn much from the behavior of the FAT states (the next great charter paper must be lurking in this data).

Here are questions for which I would love to know the answer:

  1. How do the schools that are being closed compare to district schools peers in terms of academic growth, post-secondary attainment, earnings, and parent and student satisfaction?
  2. Over a long-period (25 years?) is it better for a state to let a thousand flowers bloom and then clean up the sector or to put on tight quality controls at the outset and then  allow for measured replication? Or somewhere in-between?
  3. How does the size of a charter sector affect its political support in the state legislature?
  4. How does the quality of a charter sector affect its political support in the state legislature?
  5. How does support in the state legislature affect quality control measures?

I’m sure there is more to be mined from the behavior of the FAT states.

Brown University vs. Science

The Annenberg Institute for School Reform at Brown University just released a report on state turnaround districts.

Brown University 

The report states:

In short, the Recovery School District, which was marketed (and continues to be lauded) as ushering in a miraculous transformation in New Orleans, has not kept its promise to some of the country’s most disadvantaged students.

The report cites another report, from Stanford University, and makes the following claims about equity and accountability:

The SCOPE [Stanford center] review also found that school quality and accountability are impeded by the lack of a strong central system (within the RSD) to support instructional improvement or maintain safeguards to ensure equity and access to reasonable quality of education.


Here is what Doug Harris, who actually studied student achievement in New Orleans, wrote:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Here’s a graph that captures these gains:

Screen Shot 2015-08-04 at 7.03.28 AM


As for equity, I think this has been NOLA’s greatest innovation. I wrote a report on it. Here’s a highlight: despite serving a very at-risk student population, New Orleans has a lower expulsion rate than the state.

Screen Shot 2015-12-21 at 3.46.37 PM


As for accountability, it’s hard to think of a city that has been more serious about ensuring students don’t attend failing schools. In 2004, 60% of New Orleans students attended a school that was in the bottom 10% of the state. Now 13% do.

Screen Shot 2015-12-21 at 3.53.44 PM

People Who Live in New Orleans 

Lastly, here’s what New Orleanians think about the reforms:

Reasonable people can debate whether or not other states will see the same results.

But to say that the RSD has not kept its promise to the country’s most disadvantaged students is not supported by science.

Of course, there is still an incredibly long way to go in New Orleans. ACT scores, for example, are at an all-time high at around 19, but this still falls short of college and career ready.

Also, the New Orleans reforms were messy. While the academic results are undeniable, it’s been ten years of tense and difficult work, with many mistakes made along the way.

But if you don’t want things to be messy, you’re in the wrong line of work. The issues or race, class, and poverty are insanely complicated. If you work in the sector and haven’t changed your mind about a major issue, then you’re probably not thinking deeply enough.

All that being said… the student achievement gains are real. Children are better off.

And students across the country would be much better off if other cities achieved results similar to those in New Orleans.

Hopefully this will occur.

With the New ESSA, We’re Still Plugged into the Matrix


A great education leader who lives in Houston once said to me: “as long you’re worrying about state test scores, you’re still plugged into the matrix.”

His point: so long as public schools are held accountable via government tests, the incentives for educators will be about doing well on those tests.

If you believe performance on these tests is a useful measure of learning, then staying plugged into the matrix might be a good thing.

If you feel that parents, schools, universities, and employers are best suited to develop measures of learning, then you probably want to get out of the matrix and align incentives around different outcome measures.

In the long run, I think it’s probably a good idea to leave the matrix, so long as leaving the matrix is accompanied with a shift towards relinquishment, whereby educators can run schools and families can choose from these schools.

However, as long as we’re going to stay in the matrix, I think the two most important things are ensuring that the matrix is:

1) heavily weighted towards academic growth (rather than absolute scores) and;

2) that it identifies and acts on bottom performing schools and subgroups (where research indicates accountability helps the most).

Given that much discretion will be left to the states, time will tell if this matrix is a better than the previous matrix.

But either way, have no doubt about it: we’re still plugged into the matrix.

How I Responded on Email Chain About Rewriting NCLB

I was recently on an email chain where very smart people were debating the NCLB rewrite.
The debate had to do with whether or not the retreat from federally mandated accountability was a  good thing.
See below for my exact response:
Some thoughts:
  1. The evidence on NCLB (annual testing, data transparency, etc.) is ok but not amazing. I think the upper bound I’ve seen is .2 effects over 6 year period.
  2. We don’t yet have rigorous data on teacher evals.
  3. We do have rigorous evidence on urban charter: ~.1 effects over a 3 year period (with the sector rapidly getting better each year – effects doubled over a couple year period).
  4. And now we have rigorous evidence on NOLA charter district reforms: ~.4 effects over a 5 year period; of course under unique circumstances.
All this leads me to believe (not with absolute confidence!):
  1. The federal charter program may end up being the most important federal education intervention. Tripling it from $250M to $750M will probably do more good for low-income kids than nearly every other federal program.
  2. The testing, accountability, eval movement will likely deliver real and modest gains. But it will never change the game. I am highly skeptical that mediocre school systems get excellent due to these backend levers.
  3. The 20-50 year game, I think, is about transitioning our public operated system to a publicly regulated but non-profit operated system + better teacher pipelines + tech.
  4. This is the .5-1 standard deviation game. It’s 75% supply and at most 25% accountability.
Just some thoughts. Obviously incredibly complicated. If there was a clear answer this many smart people wouldn’t be arguing about it.
You can hear clear undertones of the Allure of Order and the New Orleans theory of change.
I view it is as a near impossibility that accountability will ever deliver transformational results.
I wish our national policy conversation was 100x more about supply and 10x less about standards and accountability.

The Next Phase of School Reform in ______________

Screen Shot 2015-07-12 at 6.03.11 PM

Richard Whitmire just had a piece in the WAPO: The next phase of D.C. education reform.

Richard’s main point is that D.C. needs the leadership and regulatory structure to reverse the chronic underperformance of many of the district’s most struggling schools.

Richard is correct.

My only objection is that this piece could have been written about just about any major urban school district in the country.

In so many cities, two things are occurring:

(1) Districts are sitting on failing schools that continue to screw over poor children.

(2) Charters are not serving every child.

To me, there is a clear solution: institute unified accountability and unified enrollment.

Judge all schools the same. All schools that continually fail should have their governance transferred to a new operator.

Give families equal access to all schools. Create fair processes for front end enrollment and back end expulsions.

And ensure that there is regulator that can impartially execute both of these functions.

This is the next phase of school reform in ________.

Stock Prices, School Letter Grades, Prediction Markets. Oh my.


Well, here’s a post that should bump up my corporate reformer ranking.

Yesterday, I read an article in the NYT titled: “Wall Street Might Know Something the Rest of Us Don’t.”

The point of the headline is that stock prices are information signals, and if stock prices are moving we should try and figure out why.

What makes stock prices particularly useful signals is that: (1) they are signals that incorporate predictions about future performance and (2) they are backed by money.

Prediction markets share these same two qualities, but are different in that (amongst other things) they don’t involve shareholder ownership of companies. You can read more about why some support the expansion of prediction markets here.

Currently, my preferred education information signal is school letter grades, which, when done well, I find to be useful in informing parents about school quality.

The weakness in letter grades, however, is that they are (1) backwards looking and (2) are not backed by money.

So here’s the question: would the existence of public school prediction markets increase student achievement in the United States of America?

You could imagine prediction markets being used in multiple ways.

1. Superintendent selection: prediction markets could be set-up where people get make bets on whether a given superintendent candidate would increase a metric such as ACT scores. If the superintendent was hired, the market would go live. This could give school boards, which far too often make overly political decisions, both information and cover when making superintendent hires.

2. Charter school authorization: same idea, but with approving charter schools – predictions would be on school performance.

3. Policy proposals: same idea, but with policy proposals (iPad rollouts, for example).

One could go on.

Some reflections:

1. Clearly, prediction market proposals would face brutal political battles. With this post, the probability that I will ever be a superintendent likely just dropped from .4% to .1%. But, whatever, ideas have to start somewhere.

2. When I led NSNO, which allocated to start-up funding to charter schools, I would have found prediction markets on charter authorization to be very, very useful. Instead of having to rely solely on on our internal diligence, we could have had access to the aggregated diligence of many others. Some schools that NSNO launched ended up failing, to the detriment of thousands of children. Prediction markets could have helped prevent this.

3. If someone vocally opposes prediction markets in any area, it’s worth asking: why? Sometimes it’s because of an emotional reaction against markets. Sometimes it’s because they really don’t care about making optimal decisions.

4. Sometimes it’s because prediction markets won’t work for practical reasons: market manipulation, lack of liquidity, and other real threats exist – though I generally think these reasons are overblown.

5. Lastly, it’s worth noting that an economics blogger is in the process of setting up a NGDP prediction market that, if successful, could change the future behavior of the Federal Reserve. So one prediction market has been launched from a blog – why not another?

At best, I think prediction markets would be of great benefit to students in our country. At worst, I doubt they’d do much harm.

Someone should try to set one up. I’d be happy to help.

HT to Robin Hanson for informing my thinking on much of this issue.

Why Letter Grades Matter

letter grades

New York City schools will no longer receive letter grades. You can read about it here.

It is likely that Chancellor Farina and I have different opinions on the role of government in schooling.

Chancellor Farina likely believes that government should operate schools and that the district’s central office should support struggling schools. In this system, developing a robust academic central office becomes a top government priority.

I believe government should regulate a system where non-profit organizations operate schools. I also believe families should be able to choose amongst these school. In this system, providing clear, transparent information becomes a top government priority. 

Of course, government can provide information in numerous ways.

Because of my experience in New Orleans, I consider letter grades to be amongst the best way to provide information to families.

When I first started working in New Orleans, schools in Louisiana were graded on a star rating system. This did not provide clear information to families because most families could not, on a gut level, distinguish between a two star, three star, or four star school.

We then moved to a letter grade system. However, we did not put a school’s letter grade on the universal enrollment form that listed all schools in the city (and which families used to enroll their children via the citywide enrollment system). As such, families had to locate the letter grade from either the citywide school’s guide or the education department’s website.

When we analyzed the data, we were surprised how many families selected “D” and “F” schools on the universal enrollment form.

We did not know if this was because families did not care about the low letter grades or because they did not believe that letter grades were a good indicator of school quality.

So the following year we put the letter grade next to each school’s name on the universal enrollment form.

When we analyzed the data this time, there was much stronger correlation between school performance (as measured by letter grades) and family demand.

So here’s what we learned. Families utilized government information on school performance when: (1) the city developed a universal enrollment system (2) performance information was provided in letter grade form (2) and letter grades were noted on the enrollment form.

All of this makes me highly skeptical that NYC’s 16-18 page information packet on each school is going to help families make good decisions. Especially when it doesn’t include any kind of final rating.

Of course, if you don’t believe that school performance can be captured in a singular rating, then my whole argument is beside the point.

But as it happens, test scores are one of the few areas of school performance that can be reliably measured and are connected to life outcomes.

As such, I think it’s one of the only metrics that government should use to grade a school. The quality of other important aspects of schooling (athletics, character building, etc.) are all best determined by families. At most, government should provide narrative descriptions of school offerings.

The fact is that if: (1) government executes a sound testing regime and (2) translates the resulting performance into a reasonable letter grade system then (3) families will get very valuable information on the quality of schools.

Without letter grades, families know significantly less about the quality of public schools.

And when families know less about the quality of public schools, they end up sending their children to schools that will negatively affect their children’s life outcomes.

This is why letter grades matter.