Category Archives: Accountability

A weak SIGnal: flawed research means we don’t know if SIG worked

I have a piece in Education Next about the study that came out the $7 billion federal school turnaround program.

Everyone is saying that the study proves SIG didn’t work.

I disagree.

My main argument is here:

In detailing these results, the authors note:

“The smallest impacts our benchmark approach could detect ranged from 0.19 to 0.22 standard deviations for test score outcomes, from 0.15 to 0.26 standard deviations for high school graduation, and from 0.27 to 0.39 standard deviations for college enrollment.”

Now, look back up at urban charter effects and you’ll see the three year results in math are about at the floor of what the SIG study could detect, and the results in reading are much lower than what the SIG study could detect (the SIG study also tracked children for 3 years).

So even if SIG achieved the same effects as urban charter schools the study may not have been able to detect these effects. 

It seems pretty unfair for charter (or voucher) champions to call SIG a failure when SIG might have very well achieved near the same results as urban charter schools.

My conclusion:

Until I see results that show that SIG worked, I won’t change my prior belief that SIG funds would have been better spent on high-quality charter growth.

Moreover, neither the existing research base nor theory warranted a $7 billion spend on district turnarounds, so even had the intervention worked I still would consider it a lucky outcome on an ill-advised bet.

But I also won’t claim that SIG failed.

Due to poor research design, we simply don’t know if that’s true.

The study authors, reporters, and commentators should walk back their strong claims on SIG’s failures.

At the same time, we should all keep advocating for government investment amounts to be in line with the existing evidence base.

If we have no reason to believe something will work, we should not spend $7 billion.

Too often, moonshots garner more status then they deserve.

Read the the whole piece here.

What Happens When What Works for Children Doesn’t Feel Good?

Closing schools does not feel good: it’s painful for families, educators, and politicians.

But closing schools, and opening new better schools, can dramatically help low-income children.

Sometimes, the best thing that can happen to a child is for her school to close.

Closing Schools Led to a +.3 SD Gain for Elementary Students in NOLA 

In New Orleans, Tulane researchers found that closing schools and creating new better schools led to very significant achievement gains:

screen-shot-2016-10-19-at-4-33-16-pm

Elementary students who attended a failing school started .1 SD behind their matched peers – two years later, these same students were .2SD ahead of their matched peers.

This +.3 SD impact is higher than the impacts of most educational interventions, and it equates to closing about 33% of the black-white student achievement gap.

You Don’t Have to Harm Existing Children for the Sake of Future Children

What’s incredible about these results is that the students whose schools were closed increased their student achievement.

Before seeing this data, my guess would have been that closing schools slightly harms existing students but is much better for future students who get to attend a better school without going through the disruption of closure.

But the NOLA data indicates that it’s possible to help both existing and future students, which should increase your belief in the benefits of school closure.

You Should Not Close Failing Schools and Send Children to Other Failing Schools 

In Baton Rouge, school closure did not lead to positive effects. This seems to be because these students enrolled into other failing schools after their original school was closed.

 

Screen Shot 2016-10-19 at 4.45.08 PM.png

For school closure to work, a city needs to either have open spaces in existing higher-performing schools or be opening new high-quality schools.

There are Good and Bad Ways to Close Schools

School closures are hard, but they can be done with respect. Families deserve to know why their schools are being closed; families should get support (and preference in unified enrollment systems) in finding a better school; and political leaders should ensure that empty school buildings are put to good use for the community.

Unfortunately, in many cities, political officials do not close schools thoughtfully. Instead of being honest with families about the poor performance of the school, they let failing schools linger for year until enrollment dwindles and the school folds academically and financially.

It is Difficult to Scale Something that Causes Political Pain

It is unclear to me whether or not deliberate school closure will scale. Reforms that cause political pain tend not to do well over time.

However, opening new great schools need not be politically painful, which bodes well for continued charter growth.

Of course, continued charter growth can lead to the closure of failing schools – and this is exactly why charter moratoriums have some political support.

Charter moratoriums have the potential to reduce pain for adults even as they inflict pain on children.

Can New Orleans Continue to Close Schools? Should It?

Over the past few years, academic performance has stagnated in New Orleans:

screen-shot-2016-10-21-at-8-20-25-am

During this time, there has also been a reduction in school closure activity.

So here is an interesting question: has the stagnation in performance been caused because New Orleans ate up most of the low-hanging fruit of closing schools, or has the stagnation in performance been caused because New Orleans has slowed down on closing failing schools?

At this point, I’m not familiar enough with the data to have strong opinions.

But I do worry that New Orleans, especially as it moves toward more local control, may stop using one of the strategies that has proven to dramatically improve the achievement of its students.

Every Mentor Should Become Your Peer

I. Rigorous Thinking and Effectiveness 

The most effective people that I’ve worked with, or engaged with online, are extremely rigorous thinkers.

This, in some sense, is satisfying.

To the extent that the connection between effectiveness and rigorous thinking is causation rather than correlation, then increasing the rigor of one’s thinking can help lead to greater effectiveness.

II. Implicit and Explicit Instruction 

Of the extremely rigorous thinkers I’ve interacted with, only some of them are good at explaining why they think the way they do.

All of them, on the other hand, have been good at telling me when my thinking has not been rigorous enough.

Given the above, I’ve spent a lot of time trying to take their implicit and explicit instruction in order to improve myself.

One incredibly positive aspect of the internet is that it opens up thousands of avenues to learn how other people think.

III. Mentorship 

Another way to say all this: the value of a mentor (either in person or online) is that she should significantly increase the rigor of your thinking.

When you communicate with her, you should run as many cycles of feedback as you can –  pitching your ideas, getting her critiques, and learning.

But, at some point, this can lead to laziness: you might rely on your mentor to do the thinking for you rather than doing it yourself.

One way to test your potential laziness is to continually take stock of the delta between the rigor of your thinking and the rigor of your mentor’s thinking. If you are not closing the gap between the rigor of your thinking and your mentor’s thinking, then you are, in some sense, exploiting your mentor.

Ultimately, every mentor should become your peer. This should be the goal of all involved.

IV. My Journey in Unrigorous Thinking 

For whatever it’s worth, my biggest obstacles to rigorous thinking have come from the following:

1. Yearning for Silver Bullets: Very few quick fixes exist for hard problems. In part, I think I was drawn to law school because of the idea that passing the right law could fix things quickly. As it happens, this is very rarely true. Hard problems tend to have decade long answers.

2. Yearning to Have My Problems Solved: Related but slightly different: when confronted with thorny day-to-day problems, I’ll sometimes gravitate toward the easiest or first solution rather than taking the time to rigorously analyze the issue. This is nothing more than intellectual laziness but it is difficult to control, especially when you’re moving fast and making a lot of decisions each day.

3. Ignoring Politics: I’ve also succumbed to spending a lot of time on ideas or programs that never had a chance of being political viable.

4. Ignoring Executing at Scale: Even ideas that might be politically viable may not be possible to scale operationally. Especially earlier in my career, I deeply underestimated execution as a limiting variable.

5. Wanting to be Agreeable: The desire to please or get along with others sometimes trumps my effort to push to get to the right answer.

Those are the big ones: I could have avoided many of the biggest mistakes of my career had I been more rigorous at avoiding these pitfalls.

I suspect I’m not the only person in my line of work to make these types of mistakes.

V. Thinking More Rigorously Hurts Until It Becomes a Habit 

For all of the above issues, there has either been a professional failure or extremely direct piece of feedback that has provided a wakeup call that I need to be thinking more rigorously.

Even after knowing I have a problem with the way I think, many times the only way I’ve been able to make progress  has been by being exposed to leaders who are adept at avoiding these pitfalls.

This often all rather painful.

For me, it literally hurts to think more rigorously in an area where I have not previously been a rigorous thinker.

In some instances it feels like being a child who knows his parents are right but doesn’t want to admit it even though it is in his best interest to (1) admit it and (2) incorporate his parent’s way of thinking into his world view.

Eventually, it gets less painful – and then it becomes a habit.

And then it’s on to the next one.

VI. Accountability is an Accelerant 

One last point: putting yourself into situations with high accountability for outcomes is one of the best ways to increase the rigor of your thinking.

If you are not accountable for outcomes, then you will be tempted to avoid the pain that comes with thinking more rigorously.

If you are accountable for outcomes, then the pain of potential failure helps offset the pain of thinking more rigorously.

The Great Charter School Clean Up?

 

Florida, Arizona, and Texas are known for having large charter school markets with large variation in quality.

Taking the first letter from each state name, let’s call these the FAT states.

All told, charter sectors in FAT states serve about 750,000 students (AZ = 180,000,FL = 280,000, TX = 280,000) – or about 25% of all charter students in the country.

Results in the FAT states have been mixed.

Here is what CREDO found in 2015 when they studied Texas charter school data:

Screen Shot 2016-02-06 at 10.13.34 AM

Even in urban areas, where charters usually perform the best, Texas charters lag compared to their traditional peers.

Previous CREDO studies in Arizona and Florida have found negative to mediocre results; however, more recent studies, especially those focusing on attainment, have found more positive results.

But, in terms of matched test results, the FAT states tend to poorly when compared to the charter sectors of Louisiana, Colorado, and Massachusetts.

___

The FAT states split the reform community.

For many of those whose posts show up on Jay Green’s blog, the FAT states are exactly what we need: high levels of entrepreneurship, disruption, and parent choice.

For many of those whose posts show up on CRPE’s blog, the FAT states have serious shortcomings: they represent the triumph of free market mania over the pragmatic restraints of quality control.

___

But something interesting is happening in the FAT states.

They are closing a lot of charter schools.

Last year, Florida closed 35 charter schools; Arizona closed 30 charters schools; and Texas closed 62 charter schools.

In Florida, regulators closed ~5% of all charter schools in a single year.

In Arizona, regulators closed ~6% of all charter schools in a single year. 

In Texas, regulators closed ~8% of all charter schools in a single year.

These rates are higher than the national charter school closure rates of ~4%.

In the case of Texas, their closure rate was double the national average.

___

We will learn much from the behavior of the FAT states (the next great charter paper must be lurking in this data).

Here are questions for which I would love to know the answer:

  1. How do the schools that are being closed compare to district schools peers in terms of academic growth, post-secondary attainment, earnings, and parent and student satisfaction?
  2. Over a long-period (25 years?) is it better for a state to let a thousand flowers bloom and then clean up the sector or to put on tight quality controls at the outset and then  allow for measured replication? Or somewhere in-between?
  3. How does the size of a charter sector affect its political support in the state legislature?
  4. How does the quality of a charter sector affect its political support in the state legislature?
  5. How does support in the state legislature affect quality control measures?

I’m sure there is more to be mined from the behavior of the FAT states.

Brown University vs. Science

The Annenberg Institute for School Reform at Brown University just released a report on state turnaround districts.

Brown University 

The report states:

In short, the Recovery School District, which was marketed (and continues to be lauded) as ushering in a miraculous transformation in New Orleans, has not kept its promise to some of the country’s most disadvantaged students.

The report cites another report, from Stanford University, and makes the following claims about equity and accountability:

The SCOPE [Stanford center] review also found that school quality and accountability are impeded by the lack of a strong central system (within the RSD) to support instructional improvement or maintain safeguards to ensure equity and access to reasonable quality of education.

Science 

Here is what Doug Harris, who actually studied student achievement in New Orleans, wrote:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Here’s a graph that captures these gains:

Screen Shot 2015-08-04 at 7.03.28 AM

Equity 

As for equity, I think this has been NOLA’s greatest innovation. I wrote a report on it. Here’s a highlight: despite serving a very at-risk student population, New Orleans has a lower expulsion rate than the state.

Screen Shot 2015-12-21 at 3.46.37 PM

Accountability 

As for accountability, it’s hard to think of a city that has been more serious about ensuring students don’t attend failing schools. In 2004, 60% of New Orleans students attended a school that was in the bottom 10% of the state. Now 13% do.

Screen Shot 2015-12-21 at 3.53.44 PM

People Who Live in New Orleans 

Lastly, here’s what New Orleanians think about the reforms:

Reasonable people can debate whether or not other states will see the same results.

But to say that the RSD has not kept its promise to the country’s most disadvantaged students is not supported by science.

Of course, there is still an incredibly long way to go in New Orleans. ACT scores, for example, are at an all-time high at around 19, but this still falls short of college and career ready.

Also, the New Orleans reforms were messy. While the academic results are undeniable, it’s been ten years of tense and difficult work, with many mistakes made along the way.

But if you don’t want things to be messy, you’re in the wrong line of work. The issues or race, class, and poverty are insanely complicated. If you work in the sector and haven’t changed your mind about a major issue, then you’re probably not thinking deeply enough.

All that being said… the student achievement gains are real. Children are better off.

And students across the country would be much better off if other cities achieved results similar to those in New Orleans.

Hopefully this will occur.

With the New ESSA, We’re Still Plugged into the Matrix

4c73266777ce9631213fa52dd8a0b855e4f9e751

A great education leader who lives in Houston once said to me: “as long you’re worrying about state test scores, you’re still plugged into the matrix.”

His point: so long as public schools are held accountable via government tests, the incentives for educators will be about doing well on those tests.

If you believe performance on these tests is a useful measure of learning, then staying plugged into the matrix might be a good thing.

If you feel that parents, schools, universities, and employers are best suited to develop measures of learning, then you probably want to get out of the matrix and align incentives around different outcome measures.

In the long run, I think it’s probably a good idea to leave the matrix, so long as leaving the matrix is accompanied with a shift towards relinquishment, whereby educators can run schools and families can choose from these schools.

However, as long as we’re going to stay in the matrix, I think the two most important things are ensuring that the matrix is:

1) heavily weighted towards academic growth (rather than absolute scores) and;

2) that it identifies and acts on bottom performing schools and subgroups (where research indicates accountability helps the most).

Given that much discretion will be left to the states, time will tell if this matrix is a better than the previous matrix.

But either way, have no doubt about it: we’re still plugged into the matrix.

How I Responded on Email Chain About Rewriting NCLB

I was recently on an email chain where very smart people were debating the NCLB rewrite.
The debate had to do with whether or not the retreat from federally mandated accountability was a  good thing.
See below for my exact response:
_________
Some thoughts:
  1. The evidence on NCLB (annual testing, data transparency, etc.) is ok but not amazing. I think the upper bound I’ve seen is .2 effects over 6 year period.
  2. We don’t yet have rigorous data on teacher evals.
  3. We do have rigorous evidence on urban charter: ~.1 effects over a 3 year period (with the sector rapidly getting better each year – effects doubled over a couple year period).
  4. And now we have rigorous evidence on NOLA charter district reforms: ~.4 effects over a 5 year period; of course under unique circumstances.
All this leads me to believe (not with absolute confidence!):
  1. The federal charter program may end up being the most important federal education intervention. Tripling it from $250M to $750M will probably do more good for low-income kids than nearly every other federal program.
  2. The testing, accountability, eval movement will likely deliver real and modest gains. But it will never change the game. I am highly skeptical that mediocre school systems get excellent due to these backend levers.
  3. The 20-50 year game, I think, is about transitioning our public operated system to a publicly regulated but non-profit operated system + better teacher pipelines + tech.
  4. This is the .5-1 standard deviation game. It’s 75% supply and at most 25% accountability.
Just some thoughts. Obviously incredibly complicated. If there was a clear answer this many smart people wouldn’t be arguing about it.
-N
_________
You can hear clear undertones of the Allure of Order and the New Orleans theory of change.
I view it is as a near impossibility that accountability will ever deliver transformational results.
I wish our national policy conversation was 100x more about supply and 10x less about standards and accountability.