The KIPP pre-K study doesn’t tell us if KIPP pre-k works

Mathematica just published a study on KIPP pre-k. The Laura and John Arnold Foundation (where I work) funded the study.

The study was well designed and asked a very important question: does high-quality charter pre-k provide benefits above and beyond simply attending a  high-quality charter elementary school?

The policy implications of this question are important: if high-quality charter pre-k adds to the achievement of students who attend high-performing elementary charter schools, policies that support both early child funding and charter school growth could accelerate achievement gains.

However, if attending a high-quality charter pre-k has no effect beyond the impact of simply attending a high-quality charter elementary school, then our limited public resources may be best spent on expanding high-quality charter elementary schools and not devoting as much resources to pre-k.

In other words, it’s really important to understand if:

KIPP pre-K + KIPP elementary > KIPP elementary

Unfortunately, due to small sample sizes as well as sample attrition, the researchers were not able to answer this question.

While the effects of KIPP pre-K + elementary were larger than the effects of KIPP elementary, these results were not significant.

The researchers write:

In reading, the magnitude of the positive impact was larger for the pre-K cohort than the kindergarten cohort on both the Letter-Word and Passage Comprehension tests administered in grade 2 (by 0.20 and 0.06 standard deviation units, respectively). Neither of these differences is statistically significant; however, the study did not have sufficient power to detect differences of this magnitude. Thus, they may be suggestive of some additional benefit in reading resulting from an offer of admission to KIPP pre-K, above and beyond the impact of an offer to KIPP in kindergarten.

In math, however, the impacts for both samples are identical, suggesting no additional benefit of KIPP pre-K beyond the impact of a KIPP elementary school.

Below you can see the estimated (non-significant) impacts:

 

Screen Shot 2017-08-23 at 12.23.34 PM.png

So, for now, we still don’t know if KIPP pre-k adds any value over KIPP elementary school.

This is no knock on KIPP or the researchers, it’s just the consequence of the limitations of working with small sample sizes that are subject to attrition.

That being said, I do think some of my friends are playing a little fast and loose in over interpreting this study, and I think we all need to be honest about the fact that we don’t yet know if KIPP pre-k “works.”

Phase II of the charter and choice research agenda is extremely important

In a recent post, I summarized much of the research on charter school and high choice cities.

The short of it: there’s a strong body of evidence that urban charter schools outperform traditional schools, and a nascent body of evidence that thoughtfully implemented high choice cities can deliver strong achievement gains.

It’s worth taking a moment to celebrate this: the research on urban charter schools is an impressive body of work built on twenty years of studying the incredible efforts of educational entrepreneurs. It’s wonderful to see these schools working.

___

But as charters scale, additional questions need to be studied.

More specifically:

  1. Do urban charter school students achieve better long-term life outcomes?
  2. What happens to traditional schools when charter schools expand?
  3. What happens to students when under-enrolled traditional schools eventually close?
  4. What happens to cities that transition to majority charter systems?

A few recent studies have shown that charter school students achieve better long-term life outcomesthat charter school expansion can improve traditional schools; that student attending schools that are closed actually benefit from the closure so long as good new schools are being continually opened; and, as noted in the previous post, that majority charter cities can deliver strong achievement gains.

But there are also studies that indicate the opposite. Charter students in Texas did not have greatly improved life outcomes; students in Baton Rouge did not increase achievement after their schools were closed; and cities such as Detroit continue to struggle despite high charter penetration.

___

Personally, I most care about questions (1) and (4). If charter schools end up delivering better life outcomes, and majority charter cities work, I’d be willing to accept some interim negative effects of charter expansion and school closures. But I understand that the latter effects could cause hardship for families, and I’m heartened to see early examples that show that charter expansion can be coupled with gains for all students.

So here’s to holding out for the best scenario: I truly hope that phase II of the research agenda shows that charters increase life outcomes, that charter school expansion and failing school closure benefits all students, and that majority charter systems deliver benefits for all students.

Initial research indicates that this might be possible, which is really exciting.

Philanthropy is a low accountability sector (and what to do about it)

In terms of accountability to stakeholders for continued survival, my ranking of sectors is as follows (from most accountable to least accountable):

  1. Politics: the people you serve vote on your fitness every 2-4 years.
  2. For-profit: the people you serve must give you their money.
  3. Non-profit: the people who care about your issue must give you their money.
  4. Philanthropy: people must be willing to accept your money.

Depending on the specific office / company / non-profit, these rankings might shift a bit, but at the median I think they hold true.

If you disagree, think about this question: which sector is the most ruthless? This is likely where this is the greatest accountability.

___

At the structural level, philanthropy is an extremely unaccountable sector. All you have to do to stay in business is have someone be willing to accept your money!

This is not the fault of anyone involved, it’s just the nature of the beast.

I think a few things follow from this structural condition:

Culture matters: Philanthropists, like all humans, care about what people (especially their peers), think of them. Given that direct accountability is not as strong of a lever in philanthropy, culture pressures will play a bigger role. I think the rise in status of evidence based giving has been a positive development on this front.

Governance matters: Certain organizational structures, such as foundations whose governance outlive their founders by 100+ years, should probably limited, as this further decreases accountability in an already low accountability sector.

Rigorous exits matter: In most cases, philanthropy can’t cover the permanent costs of its subsidies, and over time government will generally pick up the tab of continued social programs. As such, it’s government that controls when philanthropy can exit, and the more rigorous government is in its spending, the more accountable philanthropy will be for producing results.

___

Again, I don’t think philanthropy is to blame for operating in it a low accountability sector.

It is what it is.

But citizens and governments can increase accountability, and they should do so when feasible.

 

 

The Atlantic vs. Facts

I’m sympathetic to how hard it is for journalists to cover complicated policy areas.

Translating policy into readable articles that both inform and entertain is difficult. In this sense, writing a policy a blog is a luxury. For the most part, I just get to stick to the research and ideas.

But journalists do have a duty to accurately cover the basic facts of the issue they are covering.

In her recent article on Japan’s education system, Alana Semuels included a throw away line about New Orleans education reforms; she writes:

The equity in Iitate stands in stark contrast to a place like New Orleans, which was also hit by a disaster. While Japan’s national government tried to ensure that students in the affected area got more resources after the accident, officials in New Orleans disinvested in the public educational system in their city. Public-school teachers were put on leave and dismissed, many students disappeared from schools’ rolls, and the New Orleans system now consists almost entirely of charter schools.

If you click on the links, which presumably justify her claims, you don’t get taken to supporting research; instead, you get taken to an op-ed.

As it happens, rigorous research has been done on the New Orleans reforms.

In terms of student achievement, after Katrina New Orleans saw some of the biggest gains ever recorded in an urban school district. You can read about the research here.

In the article, the authors of the study note:

For New Orleans, the news on average student outcomes is quite positive by just about any measure. The reforms seem to have moved the average student up by 0.2 to 0.4 standard deviations and boosted rates of high school graduation and college entry. We are not aware of any other districts that have made such large improvements in such a short time.

Rigorous research has also been done on how public spending in education changed in New Orleans after Katrina. Contrary to Semuels claim, spending went up.

The authors of the study write:

New Orleans’ publicly funded schools spent 13% ($1,358 per student) more per pupil on operating expenditures than the comparison group after the reforms, even though the comparison group had nearly identical spending before the reforms.

A quick google search could have turned up both of these studies.

I wish Semuels had taken the time to review these studies.

Because she did not, thousands of readers have been misinformed.

Does it matter if charter schools make traditional schools better?

A new study found that traditional schools in New York City improved when charter schools opened up in the near vicinity. See this Chalkbeat article for coverage.

The effects were small for neighborhood impact (traditional schools improved +.02 SDs) and larger for co-locations (traditional schools improved +.06-.09 SDs).

What is the value of charter schools?

Charter schools could help students in two ways: (1) directly serving students and (2) increasing the performance of other schools.

I don’t believe that #2 must be true in order to support charter schools. The very fact that charter schools help the students they serve could be enough to warrant support.

An even more complicated question: what if charter schools increased the performance of the students they served but decreased the performance of traditional schools?

If, like me, you believed non-profit operation of schools should overtime supplant direct government operation of schools, then you still might support the expansion of charter schools. The negative impact of charter growth on traditional schools, while painful, could be a price worth paying if more children were benefitted by the new system over the long-haul.

Be careful how you justify your opinions 

The reason I make this point is that I don’t think we should tie the value of charter schools to their impact on nearby traditional schools.

While this impact is something we should understand – and if in some instances there are negative impacts we should we work to reduce them – ultimately, the fact that their might be a better way to do public education should be our Northstar.

Building the new system is more important to children than protecting the old system.

But hopefully we continue to find more evidence of win / win

That being said, this study is good news: it’s pretty amazing that New York City has grown a high-quality charter sector and that this sector has helped improved traditional schools.

Hopefully future studies will show the same.

But they might not. And expanding great public charter schools might still be the best thing for children.

So we should be cautious in justifying the existence of charter schools in how they impact traditional schools.

Ultimately, their highest and best use is likely to be in the creation of a better way to do public schooling at scale.

Could the Earth ever become a Dark Forest?

In the trilogy The Three Body Problem, Liu Cixin builds his novels around the idea that the universe is a Dark Forest – i.e., when you’re moving through a dark forest and you hear the rustling of leaves, the optimal reaction is to shoot first.

More fully, the Dark Forest theory of the universe is built upon these first principles:

  1. The primary goal of each civilization is to survive.
  2. There are finite resources and space in the universe.
  3. Civilizations tend to expand.
  4. Civilizations tend to advance technologically.
  5. You have no way of truly knowing whether an alien species is peaceful or hostile.

So, if you detect an alien species – what do you do?

Under the Dark Forest theory, you kill them.

The reason you kill them is that even if they’re not hostile now, at some point they will want to survive, need more sources, and have advanced technology – which means they might just kill you.

____

I have no idea if the Dark Forest theory accurately describes the first principles of the universe.

But it made me think about something we might be able to understand with greater precision: could the Earth ever become a Dark Forest?

____

Right now, the Earth is not a Dark Forest largely because of nuclear deterrence.

North Korea might very well recognize that the existence of the United States will likely bring down their regime at some point, but they can’t act on this knowledge because we could respond to any nuclear attack with an attack that wipes them out.

Even for more robust nuclear powers, each side must live with the fact that a massive nuclear war could destroy all of humanity.

Culture also acts against the Earth becoming a Dark Forest. The scaling of large societies has been in part been sustained through cultural evolution: we now identify with nation and world instead of just kin, which, presumably, partially mitigates the 5th aforementioned principal (lack of trust).

But these conditions are not immutable: so it’s worth considering, how could one sided deterrence, assured mutual destruction, and trust…. end?

___

Unfortunately, it’s not hard to describe a scenario:

  1. There is a shortage of an important resource that is necessary to a nation’s survival, which makes securing that resource more important to a nation’s survival than the benefits of trade.
  2. This shortage, as well as the already significant cultural differences between existing rival nations (such as USA and China), erode trust.
  3. Technology advances in a manner that allows a nation that launches a first strike to kills all other humans, not allow for a return strike, and preserve the attacking nation.

How about this: there’s a water shortage that fuels nationalism, that leads to rising animosity between populous nations, and then one of them develops a synthetic virus that instantly kills all humans that haven’t received the vaccine – a vaccine that the attacking nation released in their own water supply the week before launching the attack.

___

I’m not in expert in these issues, so maybe I’ve gotten much wrong.

But if the universe can become a Dark Forest, the Earth probably can too.

If this is true, we’ll need a deterrence system for whatever set of weapons come after nuclear warheads.

But what are the odds that for every new weapon we develop we’ll also near simultaneously have an equally strong deterrence system?

They don’t seem high.

___

Please do let me know where my logic is off.

When is advocating for a diversity of school models immoral?

In many cities I work in, education reform leaders bang the drum of needing more diversity in school models in their cities.

“Choice isn’t choice if all the schools are the same,” is a common statement.

I get where these folks are coming from, and I hope the breadth of effective school models continues to grow. There are still too many kids who don’t thrive in the schools that exits today, and I’m glad there are entrepreneurs who are developing new models, as well as investors and intermediaries that are supporting them.

___

At the same time, in many cities, two conditions are present:

1) Tens of thousands of children are attending failing schools.

2) Effective high expectations / high support (formerly called No Excuses) schools have waiting lists.

In other words, families are stuck in failing schools and existing high-performing schools could expand to help them.

Of course, just because these conditions exist doesn’t mean that it’s immoral to invest in new school model design.

But the “choice isn’t choice” refrain feels pretty privileged.

When your child is in a decent school, diversity of school model might be exactly what you’re looking for in a better option.

But when your child is in a failing school, you’re often just looking for an effective school who will nurture your child’s academic and personal growth – and if that school already exists in your city, you’re simply trying to get in.

___

To put a rough marker on there, if the aforementioned conditions exist in your city (many students in failing schools, many good schools with waiting lists), I think you should be devoting a good ~80% of your philanthropic funds on getting kids out of failing schools and expanding schools that are working.

If you’re allocations are reversed, and you’re spending the majority of your resources on new school models, I think you’re actions are not in the best interests of the children who are being harmed in failing schools.

If I had to argue against myself, I’d say that increasing the diversity of school models will increase the diversity of parents sending their children to public charter schools, which will strengthen the pro-charter political coalition. I’d also argue that new entrepreneurs and new models might pressure the incumbents to continue to adapt. Moreover, I’d argue that a real new school model breakthrough might prove to be more scalable than existing models, so investing in new models might help more kids sooner than scaling existing models. Lastly, I’d argue that all children, not just children in failing schools, deserve great school options.

I do agree with these counterarguments, but, for me, the near term weight of the moral good still sits with helping the children who are currently stuck in failing schools.

Hence my 80% / 20% calculation.

So yes, let’s keep on trying to develop new school models, but let’s make sure to check the privilege of making this argument with too much force, especially in cities where existing good schools have room to serve more kids who are stuck in terrible situations.