Jay Greene had a series of posts on choice regulation over at his blog.

His overarching argument: regulating school choice does more harm than good.

Broadly, I think Jay makes a number of good points. I also think he overstates his case.

More specifically, I think his arguments are somewhat strong on performance and pretty weak on equity.

I also think that Jay could be more conservative on how he generalizes fairly narrow research findings, especially given how hard he is on others who misuse research!

Overall, Jay made me think harder about how philanthropists should allocate resources across choice interventions. He might be right that there is too much attention to charters. I think if voucher proponents were more serious about equity regulation they could help shift the focus. I’d be happy to work with Jay and others on this. Nevada and other pilots that attempt to achieve scale could be fruitful places to partner and learn more.

Below I tackle his main points.

Government funding does not require performance oversight.

Jay notes that cash transfer programs (such as Social Security) do not come with government performance contracts. This is true.

Jay also notes that food stamps don’t come up with performance contracts. In his words: “When the government provides food stamps it does not require recipients to submit BMI measurements or other indicators of adequate nutrition.”

This is true, but it’s not a perfect example, as the government does require the providers of food to meet performance input targets (food must have some nutritional value). The government does not trust the consumer to make his or her own decisions, so choice is restricted based on the nutritional performance of the food provider’s product.

While there is not accountability for outcomes, there is heavy regulation of inputs.

More broadly, we are seeing more and more performance accountability in government health programs (Obamacare looking at things such as readmission rates) and government post-secondary programs (requiring schools that receive Pell grants to achieve certain performance targets).

Jay overstates his case when he says: “Repeating that government funds require accountability to the government is just mindless sloganeering, not an accurate description of how government programs typically operate.”

Yes, education testing is probably on the more regulated end of pay for performance. But the issue is a complicated one, and most government programs (except for direct cash transfers) regulate inputs and many also regulate outputs.

There is a vast academic literature on these various approaches.

Whether or not to regulate for outputs is worthy of deep debate, not outright dismissal.

Personally, I’m probably closer to Jay on this than the average charter supporter. I think that, over time, not regulating for performance would likely work out. In my previous writings, I’ve often said that I’d make the “all choice for no testing” trade with the far left and the far right.

But I think the issue is complicated. And, as I note below, I’m fairly comfortable with the government putting in a performance floor and closing down the very worst performers.

Test Scores are Limited Tools; Attainment is a Better Proxy for Quality than Test Scores; Vouchers Do Better on Attainment

Jay rightfully points out that test don’t measure everything: things like grit and conscientiousness are likely very important to lifetime outcomes and it is very difficult to capture these in tests.

Matt Barnum did a good response on this. In short, numerous studies have tied increases in test scores to increases in long-term outcomes. None of these studies are a slam dunk, and causation can be tricky, but there’s enough here to make a case that achievement tests measure some of what we want schools to be doing.

For these reasons, I think there’s a reasonable argument to be made that performance accountability, if it is to be used, should be used as a floor rather than the end all be all of school grading. Perhaps “A-F” systems should just be “F” systems.

Jay then argues that because testing doesn’t measure everything, attainment is actually a better measure of school quality.

Jay might be right, but it’s complicated. Yes, achieving a degree (be it high school or post secondary) will likely increase a student’s life outcomes. But the more this degree attainment is divorced from knowledge attainment (and test scores), the weaker this effect might become over time (unless employers really just care about conscientiousness, which may very well be the case).

All told, the international evidence on “schooling ain’t learning” is robust.

That being said, in terms of policy, I don’t really have much to disagree with here. As I noted above, I have mixed feelings on performance accountability as it is, and an increased focus on attainment might be very healthy for the charter community.

Jay then makes the case that vouchers do better on achievement than charters. Jay cites three studies, two of which find higher degree attainment. The Milwaukee study found increased high school degree and college persistence with voucher students, while the DC study found voucher students had increased high school graduation. The NYC programs only increased college attendance (I didn’t see any evidence on graduation). Moreover, the NYC program was only a partial scholarship, which as far as I can tell required families to pay for a portion of schooling (which makes it hard to generalize to families that could not afford to pay). The DC voucher program was fairly small in scope as well.

This does not seem to be an evidence base from which one can make strong, generalized claims about what voucher effects would be on a systems level. Only the Milwaukee and DC programs come near providing a full voucher program, and only the Milwaukee program got to real scale.

As Jay points out, the most rigorous charter research finds positive effects on test scores rather than attainment. But the evidence on test score gains is massive: CREDO has studied dozens of cities and have found an overall effect of ~.1 for urban areas serving hundreds of thousands of students. This research includes three markets where charters serve roughly half of all students or more (NOLA, DC, Detroit). And the impact on New Orleans attainment has been very significant (high school graduation rates are up twenty points). Of course, systems level evidence is  not randomly controlled, but this doesn’t mean that it’s not useful; in many instances, it’s probably more predictive than small RCTs.

If I had to bet on which intervention is most likely to work at scale, I’d be inclined to bet on a massive data set that found positive effects on test scores rather than a very narrow data set of three studies where only two study found higher degree attainment.

Of course, I might be wrong. But Jay has surely not proven his case. The evidence he cites covers small scale studies that make it difficult to generalize.

We Should Not Regulate Choice Programs for Equity

Jay first notes that heavy regulations –  “such as mandating that schools accept voucher amounts as payment in full, prohibiting schools from applying their own admissions requirements” – decreases the number of participating private schools.  This is clearly true.

Undoubtedly, more private schools will enroll voucher students if they can screen out students with behavioral issues and very low income students (which is what admissions requirements and pay-sharing would do).

In arguing against regulation, Jay writes: “But real education reform requires using the power of choice and competition to provide incentives to create more good and to reduce bad.”

Yet Jay’s version of education reform would clearly incentivize schools to not enroll students with behavioral issues or children from very low incomes. Why enroll a student who is hard to serve or who can only pay partial tuition?

This is the biggest flaw in Jay’s argument.

Jay is probably right that the performance market would correct itself over time.

It is very unclear to me that this would be true for equity.

It was surprising to me that Jay did not even find this obvious rebuttal worth mentioning.

Voucher proponents such as Jay would do well to think hard about reasonable equity regulations that ensure that decentralized choice markets offer good educational opportunities to all students.

Charter markets are moving toward unified enrollment and expulsion processes as easy to regulate for equity. Perhaps these processes are too heavy handed.

But I think some equity mechanisms need to be put in place.

Of course, you can make the argument that it is ok to have unequal public educational access so long as this system lifts all boats in the aggregate. Given that I believe that equity in access is a principle in of itself, it would take fairly large overall achievement gains for me to be willing to compromise so hard on equity.

Concluding Thoughts

There does seem to be enough in theory and evidence to support larger scale pilots of voucher programs.

This is why many people, charter school supporters included, are eager to see what we learn from Nevada.

For whatever it’s worth, my two major critique of the Nevada voucher were that the voucher amount was not enough money and that equitable access safeguards do not seem to be in place.

But overall I think it’s a very important breakthrough for choice.

So if the goal is more experimentation at scale, sign me up.

But in this experimentation I’d just argue for some basic equity guardrails so that choice is available to everyone, including the most disadvantaged.

Lastly, thanks to Jay for putting forth a good argument. It’s well worth grappling with.