Category Archives: Research

More Money or More Charter Schools?

I review some of the recent research in a post at Education Next.

Here’s some math from the post:

Increasing Funding by Even 10% is Insanely Expensive

Consider a hypothetical town with 50,000 students, all of them who are in poverty, and a per-pupil allocation of $10,000.

Over ten years, increasing per-pupil by 10% will cost the town a half a billion dollars.

To put the costs in context: on average, it costs around $1,000,000 to launch a new charter school that serves 500 students.

This puts the cost of the charter intervention at roughly $100,000,000.

Also: the charter costs are one-time costs.

So over a ten-year period, the total bill for increasing funding by 10%: $500 million.

The total cost for scaling urban charters to serve all 50,000 students: $100 million.

For a fifth of the cost, you probably get 3-5X the achievement impact.

Do read the whole piece.

A weak SIGnal: flawed research means we don’t know if SIG worked

I have a piece in Education Next about the study that came out the $7 billion federal school turnaround program.

Everyone is saying that the study proves SIG didn’t work.

I disagree.

My main argument is here:

In detailing these results, the authors note:

“The smallest impacts our benchmark approach could detect ranged from 0.19 to 0.22 standard deviations for test score outcomes, from 0.15 to 0.26 standard deviations for high school graduation, and from 0.27 to 0.39 standard deviations for college enrollment.”

Now, look back up at urban charter effects and you’ll see the three year results in math are about at the floor of what the SIG study could detect, and the results in reading are much lower than what the SIG study could detect (the SIG study also tracked children for 3 years).

So even if SIG achieved the same effects as urban charter schools the study may not have been able to detect these effects. 

It seems pretty unfair for charter (or voucher) champions to call SIG a failure when SIG might have very well achieved near the same results as urban charter schools.

My conclusion:

Until I see results that show that SIG worked, I won’t change my prior belief that SIG funds would have been better spent on high-quality charter growth.

Moreover, neither the existing research base nor theory warranted a $7 billion spend on district turnarounds, so even had the intervention worked I still would consider it a lucky outcome on an ill-advised bet.

But I also won’t claim that SIG failed.

Due to poor research design, we simply don’t know if that’s true.

The study authors, reporters, and commentators should walk back their strong claims on SIG’s failures.

At the same time, we should all keep advocating for government investment amounts to be in line with the existing evidence base.

If we have no reason to believe something will work, we should not spend $7 billion.

Too often, moonshots garner more status then they deserve.

Read the the whole piece here.

The Current Brutal Reality of Education Reform and Wage Growth

edu wealth

Going from 16K to 18K in Annual Wages 

Last week, I did a post on Fryer and Debbie’s excellent new study on the Texas charter sector.

I emailed the authors about my hypothesis that the growth of high-quality charters – even if they aren’t that much better than average traditional schools – could still be of great value if these new charters displaced chronically failing schools.

Roland was kind enough to respond but pushed that even if my hypothesis is true, the story still might be a depressing one.

His point: the numbers from his study indicate that even if we replaced all these failing schools with high-performing charters, we’re still only talking about ~1-2K in extra earnings per year for these students.

Given that many of these students end up lower income brackets, this might mean going from 16K to 18K a year in annual salary. Hardly game changing in terms of life outcomes – and surely not a ticket to the middle class.

Confronting this Potential Reality

When a study tells you what you don’t want to hear, the first reaction is often to not deal with it (in some ways I did this in my previous post).

So everyone in education reform needs to deal with this potential reality: there is some possibility that the best that education reform has to offer can only, on average, move a student from 16K to 18K a year.

Of course, this is only one study of one state. We don’t yet know if these numbers will hold under different contexts, methodologies, or timeframes.

But, at the very least, your belief that a great school can radically increase wages should be a little lower after reading this study.

Other Considerations

I’m still mulling this over, but in conversations with Roland and folks I work with, certain ideas bubbled up:

The data doesn’t capture recent improvements: A lot of the best charters have only really started focusing on college and career over the past 5 years or so. As such, the students who received the full suite of redesigned high schools, counseling, and career support aren’t represented in this study. To the extent you believe the best charters are problem solving machines, you might believe this to be true.

The work is generational: Perhaps reformed schools can only, on average, push students who would have been in deep poverty to achieve average poverty / lower middle-class status. And perhaps their children, who will grow up in better educated environments, will the be ones  to more fully make it into the middle class. But this story could be unwound through raised expectations: if we told kids they were going to make it to the middle class, and they don’t, how will they react?

Colleges are the bottleneck: Perhaps these real gains in learning are being wasted by ineffective two year and four year colleges – and that without higher education reform we won’t be able to translate K-12 gains into wage increases.

Society is tough: Just because you’re better educated doesn’t mean you can overcome racism, lack of social capital, and an over-reliance on signaling.

More interventions are needed: Great schools can’t solve everything; interventions that work on family poverty, health, and parenting are needed for schools to really move kids as far as they need to be moved.

The schools aren’t really that good: A bunch of teaching to the test just jacks up crystallized knowledge but doesn’t really give kids the human capital qualities they need to succeed in the workforce.

What Do you Do in the Face of Ambiguity?

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

An Alternative Interpretation of the Fryer / Dobbie Texas Charter School Study

Screen Shot 2016-08-18 at 10.26.46 AM

Roland Fryer and Will Dobbie just published an excellent study on the Texas charter school sector.

But it’s unclear to me that they captured a very important implication of their research.

I. Study Overview

The study found that charter schools in Texas, on average, have no impact on test scores and a slightly negative impact on earnings.

More interestingly, the study found that No Excuses charter schools increase test scores but only have a small and statistically insignificant impact on earnings.

Their paper ends with this cautionary statement:

Screen Shot 2016-08-18 at 10.34.38 AM

II. Walking Through Low Effect Size and High Effect Size Schools 

The famous Anna Karena quote goes something like this: “Happy families are all alike; every unhappy family is unhappy in its own way.”

I think the opposite is true of schools.

When I visit low effect size schools, I am often saddened by the level of dysfunction. Students walk the halls aimlessly, teachers seem woefully unprepared for working in a low-income environment, and the principal generally spends her day putting out fires.

When I visit high effect size schools, I’m often struck by how different they are. While most hit the basics of a calm culture and thoughtful instruction, they vary greatly in atmosphere, curriculum, and staffing models – as well as the overall student experience. A Summit school is very different than a Collegiate Academies school, despite both achieving high effects. Even No Excuses schools can feel fairly different from each other, though they do tend to gravitate around some core practices (that Fryer has helped illuminate).

I also think I would struggle mightily in a blind walk through of .1 and .2 effect size schools; it is highly unlikely I would be able to tell you which school has which effect.

So while it’s easy to identify schools that are a total mess, it’s a little difficult to tease out what’s going well in non-dysfuctional schools, as well as to distinguish between high-performing and very-high-performing schools.

III. Bad Schools Have Bad Effects on Earnings, Good Schools Have Neutral Effects on Earnings

I found this to be the most interesting chart in the study:

Screen Shot 2016-08-18 at 10.33.43 AM

What you see here is that going from (-.2) to (0) effect really matters for earnings. This is indicated by the rising slope in the bottom left quadrant.

Interestingly enough, once you hit (0) effect, going to (.2) effect has little effect on earnings. This is indicated by the relatively flat slope in the the bottom right quadrant.

In short, getting rid of bad schools could have a major effect on the earnings of graduates in an education system (assuming our economy is not a zero sum signaling game).

In a sense, this fits my experiences in spending time in schools. It’s very easy to see how a totally dysfunctional environment could negatively impact students, whereas it’s a little more difficult to tease out the additional impact on students once the basics are in place.

IV. Portfolio Management: What Happens When Charter Schools Grow?

In a world where states and districts are managing their portfolio of schools, the growth of functional schools will be accompanied by the phasing out of dysfunctional schools.

In the best possible world, the growth of new effective charter schools will be accompanied with a reduction in under-performing traditional and charter schools.

Overtime, a system can potentially rid itself of failing schools.

This is what happened in New Orleans.

Screen Shot 2016-08-18 at 12.37.37 PM

While the above analysis is more weighted toward absolute scores (rather than effect sizes), my hunch is that the story would stand with effect sizes as well (I have not run this data yet).

I think much of New Orleans’ gains were driven by the phasing out of failing schools.

It is much less clear to me that schools in New Orleans, to date, have figured out to crack the code of creating schools that are radically superior to your average functioning traditional school.

Hopefully they will.

V. No Excuses Charter Schools May Allow Us to Eliminate Failing Schools and Raise the Aggregate Earnings of Low-Income Students in the United States 

So another way to interpret this study is that the growth of No Excuses charter schools could be the key to eliminating failing schools and raising wages of low-income students who would have otherwise have attended failing schools.

Two things would have to hold true for this to be the case: (1) government action or family choice lead to the phasing out of failing schools and (2) No Excuses schools can maintain their neutral effects on earnings even if they enroll the most challenging students from the phased out failing schools.

In other words, for now, the importance of charter school growth might be much more directly tied to eliminating failing schools rather than vastly outperforming functional district schools.

If this is right, No Excuses charter schools might still very well be the most important education reform of the past quarter century.

Can We Raise IQ Through Schooling?

Whether or not schooling can increase fluid intelligence is perhaps the most important research question – and instructional challenge – in the education sector.

The reason the question is so important is as follows:

(1) Increased IQ is connected to numerous positive outcomes.

(2) IQ is based on a combination of crystallized and fluid intelligence.

(2) We have evidence that schooling can increase crystalized knowledge.

(3) We have much less evidence that schooling can increase fluid knowledge.

(4) Figuring out how if / how we can increase fluid knowledge will be very important to continuing to raise IQs.

___

The benefits of increased IQ seem to be numerous.

As a recent post from the Atlantic noted:

IQ correlates with chances of landing a financially rewarding job. Other analyses suggest that each IQ point is worth hundreds of dollars in annual income—surely a painful formula for the 80 million Americans with an IQ of 90 or below… Studies have furthermore found that, compared with the intelligent, less intelligent people are more likely to suffer from some types of mental illness, become obese, develop heart disease, experience permanent brain damage from a traumatic injury, and end up in prison, where they are more likely than other inmates to be drawn to violence. They’re also likely to die sooner.

Garret Jones also covered what a nation’s collective of IQ can mean for well-being:

Jones_26_1

This is not to say that having a high IQ is necessary to lead a meaningful life; rather, it’s only to say that increases in IQ are at the very least correlated with positive outcomes, and this is especially true at the societal level.

___

Previous research indicates that it’s difficult to increase fluid intelligence through schooling.

For example, research on the best charter schools in the country point to the fact that it’s easier to achieve gains in crystalized knoweldge than it is for fluid knowledge:

Screen Shot 2016-06-22 at 9.21.43 AM.png

Other research generally reflects this result: schools are more likely to deliver gains in crystallized rather than fluid knowledge.

__

With this context in my mind, I was excited to see Scott Alexander and Tyler Cowen blogging and tweeting about a new study from Sweden – a study that claims that an increase in years of schooling raised IQs for low-income / farmer families.

Here are the effects they found on IQ and EC (emotional control) for an additional year of schooling; the extension in schooling was rolled out across the country in phases which allowed for quasi-experimental analysis:

Screen Shot 2016-06-22 at 10.50.37 AM.png

As you can see, the IQ effects are the largest for children of farmers and manual laborers.

Interestingly enough, the extra year of schooling reduced emotional control across most classes; the authors posit that this might be because education instills less emotional control than actually working, and that the rolling out of an additional grade might have stressed the educational system and negatively impacted school culture.

In detailing the IQ increases, however, the authors of the study do not directly comment on the whether or not the IQ gains were achieved through increases in crystallized or fluid intelligence.

They do describe the test used, but they are not clear about which sections tested which types of intelligence.

Intelligence was measured at conscription with four sub- tests: A) Instructions, 40 items measuring verbal ability (e.g. ‘strike the fourth number, put a ring around the se- cond’); B), Concept discrimination, 40 items measuring verbal and reasoning abilities in which the task is to choose the one of five concepts that does not belong; C) Paper form board, 25 items measuring visuospatial ability in which the task is to pick one of four sets of pieces that can form a given figure (a variation of the Minnesota Paper Form Board);35 and D) Technical comprehension, 52 items (a figure is shown and questions about a technical problem asked).

Given that it seems that the test covers some crystalized knowledge (technical comprehension – perhaps?), it’s difficult to say whether or not the gains in IQ reflect gains in fluid intelligence.

___

Without a clear break down of the score changes across various components of the intelligence test, the study is hard to interpret.

We already know that schooling can increase crystallized knowledge; if this is what actually happened here, then we’ve simply added to a robust body of exiting evidence.

We don’t know whether schooling can increase fluid knowledge; if this is what actually happened here, then it’s a pretty big deal.

The Answer is 6.7 Miles. What is the Question?

Screen Shot 2016-06-15 at 8.01.52 PM

The question is: how far, on average, would a family send their child to attend a school that is in the highest category of the state accountability system compared to a school in the lowest category of the state accountability system?

This is from a recent report on the DC public school system. The analysis, while useful, isn’t perfect in that it only includes families who utilized the enrollment system, but it does add to the emerging literature on the revealed preferences of families that participate in transparent enrollment systems.

 

___

Here’s another answer: it increases racial integration.

The question is: does DC’s unified enrollment system increase or decrease racial segregation?

Screen Shot 2016-06-15 at 8.06.39 PM

Shockingly enough, assigning families to neighborhood schools that are zoned by property values is not a great way to decrease segregation.

 

___

Answer: Unclear.

Question: Do parents care about a school’s academic growth (as opposed to absolute test scores)?

Screen Shot 2016-06-15 at 8.20.46 PM

Interesting but not shocking. Parents probably care a lot about peers and status.

Also interesting, this seems more true of low-income families:

Screen Shot 2016-06-15 at 8.25.26 PM

This raises an interesting question for policy makers: given that growth more accurately measures a school’s impact, should they design grading systems that prioritize growth (as DC’s charter framework does) even though low-income parents might care more about absolute scores?

Or perhaps not – maybe low-income families aren’t considering the growth based performance framework because the government is hiding this information:

Screen Shot 2016-06-15 at 8.27.05 PM

___

One last answer: Families who aren’t assigned to a school in the lowest performance category, as well as the politicians and superintendents who seek their favor.

The question: who loves neighborhood schools?

It remains shocking to me that public leaders in cities such as Oakland are vehemently opposed to unified enrollment on the grounds that such systems will undermine public education.

The only thing a unified enrollment system undermines is the privilege of those who benefit from institutional racism and widespread income inequality.

 

 

Is Roland Fryer Right? Or has the RCT Fallacy Reared its Ugly Head?

Screen Shot 2016-04-11 at 9.34.05 AM

Roland Fryer just published a compilation guide to 196 RCTs in education. HT to my colleague Stuart Buck for passing it along.

The compilation is a good review of a bunch of interesting studies. Roland’s contributions always make me think. He also won the John Bates Clark Medal, which is basically the Nobel prize for economics for people under 40.

Yet, while this RCT compilation is informative, I’d be very, very, very hesitant to pass a bunch of laws and regulations based on this type of meta-research.

___

Increasingly, policy makers and pundits are using RCT evidence to make policy. This is generally a step in the right direction, and it’s great to see evidence playing a bigger role in policy making.

Yet, sometimes RCTs are more about Rigorously Contorted Tales than Randomized Controlled Trials.

Call it the RCT Fallacy.

In statistical terms, the RCT Fallacy is pretty close to the concept of external validity, but I think the RCT Fallacy has a little more psychology to it.

So here goes:

The RCT Fallacy occurs when thought leaders propose adoption of policies based on the results of   RCTs so as to avoid the messiness of politics, ideology, history, psychology, and evolution.

Fryer is more balanced than most, but, in this case, I think he still succumbs to the fallacy.

___

The RCT Fallacy is grounded in the following:

  • There is an inverse correlation between the external validity of a RCT and the operational complexity of an industry.
  • If you have a RCT on your side, it’s much easier to defend yourself against being unreasonable, even if the RCT has very questionable external validity.
  • If you don’t have a RCT on your side, you can be called an ideologue even if you’re making a very well thought out case.
  • This leads to the perverse incentive of thought leaders being in a safer place trumpeting policies with modest RCT support rather than proposing solutions that are grounded in a deep understanding of systems, organizations, and humans – but which are difficult to measure with RCTs.
  • RCTs overvalue what can be measured quantitatively.
  • RCTs overvalue the worth of understanding existing best practices and testing pilots over the creation of entire systems that accelerate new best practices.
  • In complex systems with complex organizations, evolution is a  better change mechanism than running RCTs and implementing best practice adoption, especially in policy areas where some type of accountability (user choice, output measurement, etc.) can “kill off” bad ideas.
  • Quasi-experimental studies are often a better way to capture the effects of the impact of complex systems, as it is very difficult to conduct large scale RCTs on system level policy adoption.

___

In other words, RCTs will never tell us:

  • Whether democracies are better than dictatorships.
  • How to invent an iPhone.
  • Whether capitalism is better than Communism.
  • Whether single payer health systems are better than market based health systems.
  • Whether or not a start-up will be successful.

Yes, well designed RCTs can inform our decisions on the above issues, but RCTs will not provide definitive evidence on these issues.

___

Fryer’s paper ends with his summary of the RCT evidence in education.

He argues that RCTs have demonstrated that four interventions work:  pre-k, high dosage tutoring, managed teacher PD, and charter schools.

The paper ends with the following rally cry:

Screen Shot 2016-04-07 at 8.00.11 PM

I’m not sure courage is what we need:

Pre-K: There is pretty mixed evidence on our ability to scale effective pre-k. Fryer himself notes: “of the 64 treatment effects recorded in these randomized studies [on pre-k], 21 were statistically positive; zero were statistically negative and 43 were statistically indistinguishable from zero.”

Again, I’m not sure “courage” is the term I’d use to describe scaling an intervention that shows zero effect 67% of the time.

Tutoring: Fryer covers some high-dosage tutoring studies that show strong effects. However, the costs of these programs are sometimes upwards of 20% of total per-student spending. Moreover, there would likely be severe human capital limitations if we tried to give high dosage tutoring to all the students who needed it.

Managed Teacher PD: Fryer covers studies that show success for Success For All and Reading Recovery programs. The data seems robust and schools should surely consider adopting these programs. But here’s the thing: nothing is preventing districts from adopting these programs right now!

Perhaps either districts know something that these RCTs aren’t picking up, or perhaps districts are so poorly run that it takes a dramatic intervention to get them to adopt effective programs that have been around for 10+ years.

Charter Schools: While I clearly support charter expansion, charter RCTs often run into the issue of using lottery data which limits trials to schools that are oversubscribed (and thus creates positive bias); as such, I generally view CREDO’s far reaching urban quasi-expermintal studies to be of more use.

___

Again, I don’t mean to pick on Fryer. I’ve learned a ton from reading his research and children would be better off universities were filled with thinkers like him. His work on “looking under the hood” of high-performing charters greatly influenced my thinking on schools, as has his research on tutoring.

Moreover, it’s much better to try and build a policy regime from RCTs than from the weak theory that comes out of many education departments.

But, ultimately, I don’t think that (a) the RCTs covered in his study make a strong case for the scaling of his preferred interventions or (b) that RCTs can ever really tell us how to best design our public education systems.

I do think we should utilize RCTs to help schools make choices about which practices to adopt, but, ultimately, we should utilize theory and quasi-expermential evidence to handle the major public policy questions concerning education, which in mind have more to do with system structure than educational practice.

“We” (researchers, thought leaders, policy makers, etc.) shouldn’t be operationally scaling much; rather, we should be running experiments that give empowered educators and families more information to make great choices.