Category Archives: Federal Education Policy

Did a federal grant to turnaround failing schools in New Orleans and Tennessee work?

Leave a reply

Back when I worked at New Schools for New Orleans, we applied for a $30m federal grant to turnaround failing schools in New Orleans and scale the model to Tennessee.

CREDO just came out with a research study on our efforts. Their findings, and my analysis, are below.

The New Schools Were Much Better than the Ones They Replaced 

Here’s what CREDO found when they compared the schools we created to the schools we replaced:

In New Orleans, we replaced schools (“closing schools”) that were at 26th percentile in the state with new schools (“CRM schools”) that performed at the ~33rd percentile in the state at the end of the study.

In Tennessee, schools went from the ~17th percentile to the ~23rd percentile by the end of the study.

To quote the CREDO report: “the CRM schools in both New Orleans and Tennessee showed significantly higher academic growth compared to the Closing schools they replaced.”

Translated into days of learning, these are large effects: “Closing school students experience 63 fewer days of learning in reading and 86 fewer days of learning in math when compared to students in non-CRM schools… students in CRM schools make comparable academic growth to non-CRM students.”

The New Schools Performed About the Same as Other Schools in the City

When CREDO compared the new schools to other existing schools (rather than the failing schools they replaced), they found no statistically significant effects:

Screen Shot 2018-02-15 at 12.07.57 PM.png

In other words, the new schools that replaced the failing schools performed no better or worse than other existing schools in the city.

On one hand, this is disappointing. Our most ambitious targets included having the new schools be amongst the highest performing schools in the city.

On the other hand, this is still a major improvement: the new schools replaced failing schools and ended up achieving at the same level of most other schools in the city.

Building a System that Keeps Getting Better 

Replacing failing schools with new schools is a process, not a one-time intervention.

Ideally, a subset of the schools you created will do really well, and then, overtime, these schools will continue to grow. The ones that don’t do well will not be supported to do additional turnarounds.

Over the long-haul, gradually increasing the number and scale of high-quality school operators is more important than the average effect of the first wave of replacements.

Here’s what CREDO found across the new schools when they compared them to existing schools:

Screen Shot 2018-02-15 at 12.11.36 PM.pngIn New Orleans, 50% of the new schools had positive effects in both Math and Reading. This is really positive: half of our turnaround schools in New Orleans achieved significantly better results than existing schools across the city.

In Tennessee, only one school had positive effects in both Math and Reading, though a few other schools had positive effects in only reading.

This makes me optimistic that the school operator base in New Orleans will continue to have the capacity to replace more failing schools over time.

The early results in Tennessee are a bit more worrying on the operator quality front, and the next few years will be extremely important in ensuring that a healthy operator base emerges.

Lastly: CREDO found that replacing failing schools with fresh start schools (that opened one grade at a time) had a higher success rate than whole school turnarounds. My takeaway here is that you need a mature operator base to do a lot of whole school turnarounds, and no city had enough capacity to really do whole school at scale. In hindsight, we should have done more fresh starts and less whole school turnarounds.

Was the Effort a Success?

At the outset of the project, I remember debating with our research partners at CREDO about how to set-up the evaluation.

I argued that we should ultimately be judged on whether or not the new schools we created were better than the failing schools we replaced.

I didn’t think we should be primarily judged on whether or not the new schools were better than other existing schools that weren’t failing.

Yes, we did include language in the grant application that had goals of schools performing much better than existing schools. And as we executed the project we tried to pick school operators that we thought could deliver top tier results. Our highest aspirations weren’t met. This is disappointing, but it does not mean the project was a failure.

Rather, I consider the project to be a positive step forward in improving public education in these cities.

Making Things Better

The result of the project strikes at the heart of what’s so difficult about education reform: our aspirations for our most at-risk children are incredibly high, but making progress in creating better educational opportunities is very difficult.

In roughly a five year period, we replaced failing schools with new schools that were on average 7 percentile points higher in state performance, which translates to an extra 60-90 days of learning per year.

If the process of opening and replacement continues, what is a modest success right now may eventually become a great success.

I hope that this occurs and that New Orleans continues on its impressive track record of increasing student achievement. As a reminder, the federal grant was just one piece of an overall effort that has radically reduced failing schools in the city:

Screen Shot 2018-02-20 at 5.47.30 PM

SaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSave

Can you throw money at the problem of charter school growth? We might find out.

Leave a reply

Screen Shot 2017-04-15 at 3.54.04 PM

President Trump’s federal budget calls for over $150 million in increased spending on charter schools.

Florida’s House of Representatives just passed a “” bill that would provide $200 million to non-profit charters that opened schools in neighborhoods with high concentrations of “D” and “F” schools (the Senate has note yet voted).

Given that the rate of new school openings has dropped over the past few years, these new funds, if approved, could stimulate the pace of growth of charter schools.

Some reflections:

Will Florida funds increase growth or shift growth?

It’s unclear whether the Florida funds would increase charter growth or simply shift growth away from other states towards Florida.

My guess is that it will increase growth for two reasons: (1) it’s hard to grow across state lines so I doubt out of state operators could absorb the full $200 million and (2) if the for-profit world is any indication, other states will follow Florida’s lead to compete for great charter schools, which ultimately will create a greater pool of growth funds across the country, which should stimulate new entrepreneurs as well as provide funds for operators to grow in their home states.

How much does money incentive growth in the non-profit sector?

For management teams, growth is much less lucrative in the non-profit sector than it is in the for-profit sector. Salaries go up a bit, but not 50-100X, and there is no cashing out of equity.

There may be some status associated with winning big grants and growing, but the lack of financial incentives to individuals is probably a big barrier to successfully throwing money at the problem.

How much does money change the emotional incentives of charter entrepreneurship?

Perhaps money will have a positive causal effect through non-financial channels by changing the emotional incentives of charter entrepreneurship.

It takes a special kind of person to open a charter school when the local district, the local teacher’s union, and half your friends are telling you charter schools are destroying public education.

But what if the federal government and (eventually) dozens of states were offering large amounts of public dollars for you to open a charter school?

When the President, the Governor, and the Mayor are all asking you to grow – and putting their money up – perhaps this changes your emotional inclinations?

How much does money lower the headaches of growth?

Perhaps incentives should be thought less in terms of accusing gains and instead in avoiding pain.

Ask a charter operator what the biggest headache is for growth and facilities will inevitably be near the top of the list.

To the extent additional funds can be used (or allow other money to be used) for down payments on facilities, these funds might help stimulate growth.

My Guess

I think a state package of (1) multiple state authorized charters and (2) money for buildings would have a positive impact on growth.

Local school board authorization and lack of facility funds are huge headaches for even the most sophisticated charter organizations.

Removing these barriers would be a positive step forward.

However, I do worry that the process of lowering barriers, increasing funds, and scaling great operators will not meet the demands of the political cycle.

As this Politico piece notes, it’s unlikely that the nation’s best operators are going to immediately scale in Florida.

My recommendation to Florida would be to mimic the growth of the federal charter school program: start small, spend the funds prudently, and the increase the amount of funding available as operator capacity to growth increases.

In other words, fight for 25 years of 10% growth, not a 2-3 year moonshot.

A weak SIGnal: flawed research means we don’t know if SIG worked

2 Replies

I have a piece in Education Next about the study that came out the $7 billion federal school turnaround program.

Everyone is saying that the study proves SIG didn’t work.

I disagree.

My main argument is here:

In detailing these results, the authors note:

“The smallest impacts our benchmark approach could detect ranged from 0.19 to 0.22 standard deviations for test score outcomes, from 0.15 to 0.26 standard deviations for high school graduation, and from 0.27 to 0.39 standard deviations for college enrollment.”

Now, look back up at urban charter effects and you’ll see the three year results in math are about at the floor of what the SIG study could detect, and the results in reading are much lower than what the SIG study could detect (the SIG study also tracked children for 3 years).

So even if SIG achieved the same effects as urban charter schools the study may not have been able to detect these effects. 

It seems pretty unfair for charter (or voucher) champions to call SIG a failure when SIG might have very well achieved near the same results as urban charter schools.

My conclusion:

Until I see results that show that SIG worked, I won’t change my prior belief that SIG funds would have been better spent on high-quality charter growth.

Moreover, neither the existing research base nor theory warranted a $7 billion spend on district turnarounds, so even had the intervention worked I still would consider it a lucky outcome on an ill-advised bet.

But I also won’t claim that SIG failed.

Due to poor research design, we simply don’t know if that’s true.

The study authors, reporters, and commentators should walk back their strong claims on SIG’s failures.

At the same time, we should all keep advocating for government investment amounts to be in line with the existing evidence base.

If we have no reason to believe something will work, we should not spend $7 billion.

Too often, moonshots garner more status then they deserve.

Read the the whole piece here.

School supply or regulatory change – what should the Trump administration incentivize?

Leave a reply

screen-shot-2016-12-13-at-7-42-57-am

Andy Smarick has a good short paper out where he gives advice to the new Trump administration.

He proposes a $250 million program that would fund “high-quality, high demand, highly accountable programs.”

In short, Andy proposes a federal fun to increase the supply of things like course choice, education savings accounts providers, voucher-based private schools – as well as the creation of innovative regulatory regimes that can monitor the performance of these entities.

I really like Andy’s suggestion that the federal government continue to invest in the creation of great providers for the following reasons:

Launching new schools and courses costs money, and given that most providers are non-profit organizations, they need funding in advance of receiving regular per-pupil revenue.

The federal government lacks knowledge of what local communities need; by funding entrepreneurs, the feds are backing those who can only succeed if they meet local demand, thereby reducing the risk that funds will work against local needs.

Innovation has positive externalities: it’s well known that, in the private sector, entrepreneurs rarely capture the full value of their innovations. The same holds true in the public sector: great non-profits do not capture all of the social good they create; rather, it leaks into the full  system over time (i.e., Teach Like a Champion).

I’m a little more skeptical that the government should incentivize the creation of new accountability policies. Whatever you think of Common Core or teacher evaluations, it’s unclear that the federal governments involvement has been a long-term net benefit for the sustainability or effectiveness of these policies.

Yes, Andy’s suggestion is much less heavy-handed. He’s only arguing that the federal government should fund accountability pilots.

But I worry that this is simply reinforcing a bad habit.

My current thinking is that the federal government should focus solely on research, information transparency, supply, and civil rights.

Of these four, supply is currently the most under-appreciated and underfunded, and Andy is right to call for an increase in funding here.

Dylan Matthews’ Education Policy is Fragile

1 Reply

Screen Shot 2015-11-01 at 8.36.37 PM

Dylan Matthews is a great writer. You should read his argument against equality of opportunity.

But in his most recent piece on education, he greatly overstates his case.

Dylan argues we should:

  1. Federalize education spending.
  2. Federalize standards and assessments.

Dylan gets three things wrong:

Federalism is Anti-Fragile

Dylan thinks it would be a good idea to federalize education spending and standards + assessments. The risks here are obvious: all are eggs are in one basket and we can’t experiment. I assume Dylan likes the idea of federalizing spending and standards because he thinks this will create more equitable spending and increase the rigor of our standards.

But just because this is what Obama would do does not mean it’s what the federal government would do, either now or in the future.

What if the federal government cut education spending by 20%? Of what if it banned the teaching of evolution? Or what if it banned the teaching of climate change?

Each of these is plausible.

The point is this: in a federalized education system, one bad federal government decision could really mess up education for all of our children.

It is fragile.

The Connection Between Funding and Outcomes is Hardly Air Tight

I recently wrote about how, once you correct for income, most states perform about the same on NAEP.

These states also happen to spend vastly different amounts of money on public education.

Dylan cites one study that shows a connection between spending and outcomes. It’s an interesting study but it’s hardly a slam dunk case.

Big picture, the connection between spending and outcomes is much murkier than Dylan indicates.

50 States Allows for Experimentation and Innovation 

This is another reason federalism is important: fifty different spending patterns allows us to better test the connection between any given policy and student achievement.

So too with numerous other policies. Innovations like charter schools were pioneered by states and then other states adopted them often after watching the original pilots unfold over time.

Yes, the government can run pilot programs. But these pilots will always be connected to the federal political ecosystem, which is inherently far less diverse than the political ecosystems of the fifty states.

In Sum

I personally think that federalizing education spending and standards is a bad idea.

Yes, the federal government can and should use its education spending power to influence states, and I think a case can be made for opt-in national standards.

But having the federal government fully controlling the primary levers of public education would radically shift education in our country, I think for the worse.

Arguments for such a drastic overhaul should be couched in humility, a deep understanding of research, and a thorough exploration of the obvious objections.

I think Dylan failed to do this.

But I’m eager to hear his response. He’s a thoughtful person and talented writer, and perhaps he can make a more expanded case for having the federal government control education spending and standards.

What Will Matter 50 Years From Now?

3 Replies

Screen Shot 2015-07-23 at 5.41.51 PM

Matt Barnum, who has been doing a good job over at The Seventy Four, just wrote a thoughtful piece on Arne Duncan’s legacy.

Matt argues that Duncan should have stuck to pushing for test based teacher evals for only those teachers covered by preexisitng annual tests. I’m sympathetic to Matt’s argument, but I also haven’t spent much time thinking about this specific issue.

What I do sometimes think about this: what policies will matter 50 years from now?

This is not to say that we should only focus on policies that will have 50 year staying power, but, in expending political capital, reform longevity should be a part of the calculation.

I am skeptical that government mandated teacher evaluations will still be a major issue in 50 years. My guess is that a combination of deregulation (charters not being a part of state evaluation systems) and technological advancement (less reliance on annual tests for measuring teacher performance) will render the issue mostly moot.

If I had created Race to the Top, I probably would have focused on the following:

1. Governance: incentivizing alternative forms of governance (RSDs, alternative authorizers, etc.).

2. School Operators: increasing supply of high-quality charters, contract, and vouchers schools.

3. Teacher pipelines: creating new pipelines and reforming existing institutions.

4. Standards and Assessments: incentivizing the raising of standards and the adoption of rigorous assessments.

I think the aforementioned initiatives would all have increased the probability of increasing student achievement. I also think these initiatives would have had some staying power.

I have no idea if they would have been politically feasible to push from the federal level in 2009.

Lastly, for whatever it’s worth, I have a lot of respect for Arne Duncan. Being a cabinet secretary for eight years takes a lot of grit and passion.

What Should Congressional Republicans Do?

Leave a reply

law

Now that Congressional Republicans have majorities in the House and the Senate, they may put forth some education bills.

Given that Obama can veto whatever they might put forth, they might want some advice from someone who leans left but supports some of their education agenda. And per the image above, maybe this will get me on Jimmy Fallon.

So here goes – with the major caveat that I’m not an expert in either federal education policy or Congressional politics. But hopefully the below can spark some thinking…

Conservative Guiding Principles

I’m assuming that they anything you all put forth will:

1. Attempt to reduce the operational role of government in public schooling.

2. Not micromanage states.

3. Not increase the amount of money the federal government spends.

A Couple More Guiding Principles To Consider

1. Put forth some bills that have chance of passing. I understand the need for some symbolic stuff, but it would be nice to get a couple bills enacted.

2. Be open to the federal intervention in breaking local monopolies. Given how our education system is structured, sometimes it takes action at one level of government to break the monopoly of a lower level of government.

3. Put forth bills that actually will affect a lot of students. Programs such as the DC voucher program are important, but fighting this type of battle expends a lot of political capital for an issue that only affects a small amount of students.

Legislative Proposals

1. Overhaul Title II. A somewhat ungenerous but partially true take of Title II is that it is a 3 billion slush fund for ineffective professional development. So, first, cut the total outlay by 25%. I don’t think any students will notice. This frees up about $800 million. Thoughts on how to spend that below. Second, re-regulate Title II to require that money only be spent on teacher and school leader preparation programs that achieve results. We have some idea what works here, and these programs can be measured at scale. We have no idea what works in educator development, and it’s very hard to measure, so cut all funding to development programs. Outside of that, let the federal department of education figure out how to implement the program. Anymore and you’ll look like you’re micromanaging.

2. Triple the Amount of the Charter Bill. Get the Senate to pass a charter bill (the House already passed it). But to take advantage of this moment in time: triple the amount of money that the House committed. Instead of $300M fund the bill at $900M.

3. Double the Investing in Innovation Program, Add a New Category for Governance, and Push New Funding to States. Double i3 from $250M to $500M but push the $250M to states on a formula based on state population. Let the states control what innovations will improve education in their communities. Also, require the federal department of education to add a new category for innovations in governance, which can be used for innovations in how public education is structured (charters, vouchers, course choice, tax credits, etc). We need much more innovation here – and this type of innovation will support much of your overall agenda.

Good luck.