Category Archives: Accountability

The Great Charter School Clean Up?

3 Replies

Florida, Arizona, and Texas are known for having large charter school markets with large variation in quality.

Taking the first letter from each state name, let’s call these the FAT states.

All told, charter sectors in FAT states serve about 750,000 students (AZ = 180,000,FL = 280,000, TX = 280,000) – or about 25% of all charter students in the country.

Results in the FAT states have been mixed.

Here is what CREDO found in 2015 when they studied Texas charter school data:

Screen Shot 2016-02-06 at 10.13.34 AM

Even in urban areas, where charters usually perform the best, Texas charters lag compared to their traditional peers.

Previous CREDO studies in Arizona and Florida have found negative to mediocre results; however, more recent studies, especially those focusing on attainment, have found more positive results.

But, in terms of matched test results, the FAT states tend to poorly when compared to the charter sectors of Louisiana, Colorado, and Massachusetts.

___

The FAT states split the reform community.

For many of those whose posts show up on Jay Green’s blog, the FAT states are exactly what we need: high levels of entrepreneurship, disruption, and parent choice.

For many of those whose posts show up on CRPE’s blog, the FAT states have serious shortcomings: they represent the triumph of free market mania over the pragmatic restraints of quality control.

___

But something interesting is happening in the FAT states.

They are closing a lot of charter schools.

Last year, Florida closed 35 charter schools; Arizona closed 30 charters schools; and Texas closed 62 charter schools.

In Florida, regulators closed ~5% of all charter schools in a single year.

In Arizona, regulators closed ~6% of all charter schools in a single year.

In Texas, regulators closed ~8% of all charter schools in a single year.

These rates are higher than the national charter school closure rates of ~4%.

In the case of Texas, their closure rate was double the national average.

___

We will learn much from the behavior of the FAT states (the next great charter paper must be lurking in this data).

Here are questions for which I would love to know the answer:

How do the schools that are being closed compare to district schools peers in terms of academic growth, post-secondary attainment, earnings, and parent and student satisfaction?
Over a long-period (25 years?) is it better for a state to let a thousand flowers bloom and then clean up the sector or to put on tight quality controls at the outset and then allow for measured replication? Or somewhere in-between?
How does the size of a charter sector affect its political support in the state legislature?
How does the quality of a charter sector affect its political support in the state legislature?
How does support in the state legislature affect quality control measures?

I’m sure there is more to be mined from the behavior of the FAT states.

Brown University vs. Science

1 Reply

The Annenberg Institute for School Reform at Brown University just released a report on state turnaround districts.

Brown University

The report states:

In short, the Recovery School District, which was marketed (and continues to be lauded) as ushering in a miraculous transformation in New Orleans, has not kept its promise to some of the country’s most disadvantaged students.

The report cites another report, from Stanford University, and makes the following claims about equity and accountability:

The SCOPE [Stanford center] review also found that school quality and accountability are impeded by the lack of a strong central system (within the RSD) to support instructional improvement or maintain safeguards to ensure equity and access to reasonable quality of education.

Science

Here is what Doug Harris, who actually studied student achievement in New Orleans, wrote:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Here’s a graph that captures these gains:

Screen Shot 2015-08-04 at 7.03.28 AM

Equity

As for equity, I think this has been NOLA’s greatest innovation. I wrote a report on it. Here’s a highlight: despite serving a very at-risk student population, New Orleans has a lower expulsion rate than the state.

Screen Shot 2015-12-21 at 3.46.37 PM

Accountability

As for accountability, it’s hard to think of a city that has been more serious about ensuring students don’t attend failing schools. In 2004, 60% of New Orleans students attended a school that was in the bottom 10% of the state. Now 13% do.

Screen Shot 2015-12-21 at 3.53.44 PM

People Who Live in New Orleans

Lastly, here’s what New Orleanians think about the reforms:

Reasonable people can debate whether or not other states will see the same results.

But to say that the RSD has not kept its promise to the country’s most disadvantaged students is not supported by science.

Of course, there is still an incredibly long way to go in New Orleans. ACT scores, for example, are at an all-time high at around 19, but this still falls short of college and career ready.

Also, the New Orleans reforms were messy. While the academic results are undeniable, it’s been ten years of tense and difficult work, with many mistakes made along the way.

But if you don’t want things to be messy, you’re in the wrong line of work. The issues or race, class, and poverty are insanely complicated. If you work in the sector and haven’t changed your mind about a major issue, then you’re probably not thinking deeply enough.

All that being said… the student achievement gains are real. Children are better off.

And students across the country would be much better off if other cities achieved results similar to those in New Orleans.

Hopefully this will occur.

With the New ESSA, We’re Still Plugged into the Matrix

How I Responded on Email Chain About Rewriting NCLB

2 Replies

I was recently on an email chain where very smart people were debating the NCLB rewrite.

The debate had to do with whether or not the retreat from federally mandated accountability was a good thing.

See below for my exact response:

_________

Some thoughts:

The evidence on NCLB (annual testing, data transparency, etc.) is ok but not amazing. I think the upper bound I’ve seen is .2 effects over 6 year period.
We don’t yet have rigorous data on teacher evals.
We do have rigorous evidence on urban charter: ~.1 effects over a 3 year period (with the sector rapidly getting better each year – effects doubled over a couple year period).
And now we have rigorous evidence on NOLA charter district reforms: ~.4 effects over a 5 year period; of course under unique circumstances.

All this leads me to believe (not with absolute confidence!):

The federal charter program may end up being the most important federal education intervention. Tripling it from $250M to $750M will probably do more good for low-income kids than nearly every other federal program.
The testing, accountability, eval movement will likely deliver real and modest gains. But it will never change the game. I am highly skeptical that mediocre school systems get excellent due to these backend levers.
The 20-50 year game, I think, is about transitioning our public operated system to a publicly regulated but non-profit operated system + better teacher pipelines + tech.
This is the .5-1 standard deviation game. It’s 75% supply and at most 25% accountability.

Just some thoughts. Obviously incredibly complicated. If there was a clear answer this many smart people wouldn’t be arguing about it.

-N

_________

You can hear clear undertones of the Allure of Order and the New Orleans theory of change.

I view it is as a near impossibility that accountability will ever deliver transformational results.

I wish our national policy conversation was 100x more about supply and 10x less about standards and accountability.

The Next Phase of School Reform in ______________

Stock Prices, School Letter Grades, Prediction Markets. Oh my.

2 Replies

Well, here’s a post that should bump up my corporate reformer ranking.

Yesterday, I read an article in the NYT titled: “Wall Street Might Know Something the Rest of Us Don’t.”

The point of the headline is that stock prices are information signals, and if stock prices are moving we should try and figure out why.

What makes stock prices particularly useful signals is that: (1) they are signals that incorporate predictions about future performance and (2) they are backed by money.

Prediction markets share these same two qualities, but are different in that (amongst other things) they don’t involve shareholder ownership of companies. You can read more about why some support the expansion of prediction markets here.

Currently, my preferred education information signal is school letter grades, which, when done well, I find to be useful in informing parents about school quality.

The weakness in letter grades, however, is that they are (1) backwards looking and (2) are not backed by money.

So here’s the question: would the existence of public school prediction markets increase student achievement in the United States of America?

You could imagine prediction markets being used in multiple ways.

1. Superintendent selection: prediction markets could be set-up where people get make bets on whether a given superintendent candidate would increase a metric such as ACT scores. If the superintendent was hired, the market would go live. This could give school boards, which far too often make overly political decisions, both information and cover when making superintendent hires.

2. Charter school authorization: same idea, but with approving charter schools – predictions would be on school performance.

3. Policy proposals: same idea, but with policy proposals (iPad rollouts, for example).

One could go on.

Some reflections:

1. Clearly, prediction market proposals would face brutal political battles. With this post, the probability that I will ever be a superintendent likely just dropped from .4% to .1%. But, whatever, ideas have to start somewhere.

2. When I led NSNO, which allocated to start-up funding to charter schools, I would have found prediction markets on charter authorization to be very, very useful. Instead of having to rely solely on on our internal diligence, we could have had access to the aggregated diligence of many others. Some schools that NSNO launched ended up failing, to the detriment of thousands of children. Prediction markets could have helped prevent this.

3. If someone vocally opposes prediction markets in any area, it’s worth asking: why? Sometimes it’s because of an emotional reaction against markets. Sometimes it’s because they really don’t care about making optimal decisions.

4. Sometimes it’s because prediction markets won’t work for practical reasons: market manipulation, lack of liquidity, and other real threats exist – though I generally think these reasons are overblown.

5. Lastly, it’s worth noting that an economics blogger is in the process of setting up a NGDP prediction market that, if successful, could change the future behavior of the Federal Reserve. So one prediction market has been launched from a blog – why not another?

At best, I think prediction markets would be of great benefit to students in our country. At worst, I doubt they’d do much harm.

Someone should try to set one up. I’d be happy to help.

HT to Robin Hanson for informing my thinking on much of this issue.

relinquishment

writing about handing power back to educators and families – and other thoughts