Category Archives: Research

Does research matter?

A common critique of public charter schools is that they hurt traditional schools.

As of 2016, this questions had been studied by researchers in 16 regions. In 15 regions, they found that public charter school growth had positive or neutral effects on student learning in traditional schools. The table below summarizes the results of these studies.

Screen Shot 2018-09-25 at 7.55.15 PM

In 2016, a ballot initiative in Massachusetts called for “lifting the cap” on charter schools. In many cities in Massachusetts, it is illegal  for public non-profit charter schools to serve more students, even if parents want to send their children to these schools.

This is despite the fact that Boston is home to some of the best charter schools in the nation. You can check out this NYT article to learn more.

By a large margin, the ballot measure was rejected by voters in Massachusetts. Prominent political leaders, such as Elizabeth Warren and Bernie Sanders, came out against it.

A post-ballot analysis showed that the measure lost because many people believed that expanding public charter schools would hurt traditional public schools.

Yet, as I noted at the beginning of the post, in 2016 we had a good amount of evidence that this wasn’t likely to be true.

Research did not matter to the Massachusetts outcome.

Now a new study has come out looking at this exact issue in Massachusetts. And the researchers found that between 2011 and 2015, charter school expansion did not negatively impact traditional schools in Massachusetts (and may have had a small positive effect).

Voters in Massachusetts hold incorrect beliefs.

So now there have been studies in 17 regions, and in 16 of these regions public charter growth has had either positive or neutral effects on traditional schools.

Will votes in Massachusetts, and elsewhere, change these beliefs?

A cynic might say no. She might argue that voters don’t vote based on research, they vote based on emotion and tribal affiliation.

There is much truth to this argument.

But I have been influenced a lot by another argument. Lant Pritchett, in this podcast, argues that while a single research study rarely changes the world, hundreds of them often do.

In other words, if a bunch of studies pile up, and they all roughly show the same thing, at some point it becomes hard to argue against them. Sooner or later, you find yourself in the anti-vaccine tribe.

I think there is a lot of merit to this argument.

So while those of that support great public charter schools should develop communication strategies that pull at the heartstrings and speak to the tribe, I also think we should keep building our evidence base.

I think research does matter.

At some point, if the research studies keep piling up, they will increase the probability that the voters of Massachusetts, and elsewhere, change their minds.

Early signs that the New Orleans reforms did more than raise test scores

For the past ten years, I’ve been worried that the test score gains we were seeing in New Orleans wouldn’t lead to longer-term benefits for New Orleans students.

I worried about this both because of the research showing that increases in test scores are not always correlated to better life outcomes, as well as the fact that many colleges in Louisiana are pretty mediocre and could still fail to educate students even if they came in better prepared.

Thankfully, just published research by Doug Harris and Matt Larson provides early indication that New Orleans students have both achieved an increase test score performance *and* better post-secondary outcomes.

While it’s wonderful to see this data, we’ll continue to learn more as additional cohorts student graduate from the new public system. So far, only a few cohorts of New Orleans students attended all of their high school and post-secondary education post-Katrina.

My expectation is that the results will improve overtime, as the major high school reforms took place later in the reform effort. Good non-profit charter school operators now run many more of the high schools than they did in the few years after the storm.

Lastly, it’s worth noting that the New Orleans reforms were not a randomized controlled trial. A tragedy occurred and educators and families did their best in the aftermath. I’m not expert enough to judge the author’s methodological choices, but am eager to see other researchers weigh in on whether or not this is the best way to estimate the impacts of the reforms.

Test Scores

Across all subjects, the researchers found +.4-.6 standard deviations effects.

A rough rule of thumb is that .25 standard deviation increase equals an additional year of learning.

By this estimate, New Orleans students achieved an additional two years of learning relative to the education they would have received before the storm.

Screen Shot 2018-07-22 at 8.30.07 AM

High School Graduation

The researchers found an estimated six percentile point increase for high school graduation. While positive, this is below the gains of twenty points often touted by reformers. The researchers explain that because high school graduation was going up across the state, some of the twenty point gains would have likely occurred without the reforms.

Screen Shot 2018-07-22 at 8.32.47 AM.png

College Enrollment

The researchers found a 10-15 percentage point jump in college entry. The effect was particularly high for on-time college enrollment (enrolling right after high school), where the rate jumped from 22% to to 37%.

Screen Shot 2018-07-22 at 8.39.13 AM.png

College Graduation 

Of course, we don’t just want to see students enter college. We hope that they graduate as well.

The researchers found +4 percentage point gain for on-time college graduation (graduation within five years of enrollment), from about 10 percent to 14 percent.

Screen Shot 2018-07-22 at 8.40.27 AM

The results for staying in college for two years were higher: about +8 percentage points. So it seems likely that many students who entered college because of the reforms made it through at least two years.

But, again, these are very preliminary results. The students in the data set for college graduation experienced very little of the reform efforts. This rate should go up over time. I really hope that 14% on time graduation rate will rise significantly in the coming years.

Cost / Benefit of the Reforms

The researchers end their paper by noting that the ROI of the New Orleans’ reform efforts is higher than class size reforms and is within range of the famous Perry pre-school study (which is probably on the very high end for pre-k results).

Screen Shot 2018-07-22 at 8.55.14 AM

My guess is that the New Orleans effort is likely near the top of what we can expect for these types of city level reforms. The low baseline achievement, coupled with the speed of the post-Katrina effort, meant that the reform efforts had the potential to show large increases quickly.

But even if other cities see a ROI of 50% less, that would still make the reforms worth scaling.

But, for now, it’s worth celebrating a bit for the students of New Orleans: more of them than ever in recent history appear to be on track for choice filled and meaningful lives.

Doubling down when you’ve been wrong

From 2012 to 2016, three reforms took place in Memphis: the Gates Foundation teacher evaluation reform, the Achievement School District (ASD) charter school turnaround effort, and the district run izone turnaround effort.

In private conversations, I predicted that neither the teacher evaluation nor izone efforts would work. I thought the ASD charter effort work would work. The Arnold Foundation funded research to help us understand if our beliefs about how to help children were correct.

Five years later, the results paint a different picture than my predictions: the teacher evaluation work did not improve achievement. The ASD charters have yet to deliver results. And the izone schools have done the best.

I was wrong about the izone and the ASD.

So what to do now?

The short answer: expand non-profit governance to protect the gains of izone schools, and continue to grow the best non-profit public charters and replace the worst.

The longer answer is below.

Responding to Being Wrong

There are three ways to respond to being wrong: ignore that you were wrong, change your behavior, or admit that you were wrong but double down.

Ignoring that you’re wrong is never good.

Changing your behavior is often the most reasonable response.

Doubling down comes with substantive and reputation risk, but it can be the right thing to do.

Memphis Results 

The teacher evaluation reforms went roughly as I expected, so I won’t discuss that too much here, other than saying that my predictions were based on the belief that execution would hamper implementation in big urban districts, which seems to be exactly what happened.

Here are the results of the izone and ASD charter efforts:

 

 

This slideshow requires JavaScript.

The izone had a positive .2 stand deviation effect over five years. This is fairly good impact and it’s great to see these schools helping students.

The ASD schools had no impact.

What I Got Wrong

I made a few mistakes.

First, I underestimated the school district’s strategic and executional competency. In creating the izone, the district successfully recruited a lot of their best educators to their izone schools, and then appointed a talented leader, Sharon Griffith, to oversee these schools. This strategy (concentrating talent) and execution (selecting a great leader) worked in ways I did not expect.

Second, I overestimated the abilities of charter networks to deliver results in the first few years of the turnaround efforts. My assumption was that expert charter leaders would be able to deliver better results pretty quickly, even though they were going into very difficult situations and were often coming from out of town. Unfortunately, a few operators really struggled in the early years, with one of them, Yes College Prep, choosing to not even open up a school (this was a big hit as YES is considered to be one of the best networks in the country).

What I Would Have Done Differently 

I should have realized that great educators in the traditional system could achieve good results if given autonomy in a competitive environment. I have changed my beliefs on this issue based of the work of the izone, the work in Lawrence, Massachusetts, and my lived experience of seeing great teachers and leaders in all types of schools.

Rather than predict the izone wouldn’t work, I should have predicted that the izone might work, but advocated for non-profit governance over the zone so that it could sustain its gains. This remains my worry with the izone: that it will not be able to sustain its impacts over the next 5-10 years due to its governance structure, which makes it vulnerable to political shifts.

I also worry that it can’t really scale. Putting the best educators in the worst school is more of a one time intervention rather than a strategy that can grow over time, and it has some negative effects on the schools that lose their great educators.

Second, I should have realized that early stage charter operators (either newly started or new to Memphis) would struggle to scale through full school turnarounds in a difficult political environment. In hindsight, I should have advocated for more one grade at a time roll outs of newer operators. While this would have been more disruptive to families at the outset, it would have been better for kids over the long run.

What to Do Now

If the above diagnosis is true, it doesn’t really call for a dramatic change in strategy.

Rather, it calls for trying to get non-profit governance over the best district schools, growing the charters that are working, replacing those that aren’t, and starting new non-profit district and charter schools under more favorable conditions.

Getting non-profit governance to the best district educators would (I think) require a change in legislation.

Growing the best charter schools and closing the worst  can be done under current policy.

Evidence to Support Doubling Down

Researchers have found evidence of charter school sectors improving over time. This paper on charters in Texas found significant sector improvements, which led the researchers to write:

The findings suggest the value of taking a longer-term perspective when evaluating the impact of a major educational reform such as the introduction of charter schools, especially when the success of the reform ostensibly depends on parental decisions and market forces.

Additionally, previous research on Memphis charters found strong positive effects:

Screen Shot 2018-06-23 at 8.03.44 AM

These results mirror the results of many other urban charter sectors, which generally find positive effects.

So there is a research based case for predicting that the Memphis charters will improve over the next 5-10 years. We have already seen this with some operators, such as Aspire.

In Sum 

The research has increased my belief that great district school educators, with autonomy and support (and often pressure on local districts from the state to give this autonomy) can really help struggling students.

I still believe that these gains risk backsliding due to political shifts, so I support giving these schools non-profit governance so they can build enduring organizations (both Denver and Indianapolis have mechanisms to give district schools non-profit status).

I also still believe that growing the best charters and closing the worst is one of the best hopes for improving student achievement in Memphis.

So, in many ways, I’m choosing to double down despite being wrong, with the caveat of calling for non-profit governance for more types of public schools.

While this is uncomfortable, I don’t know a better way forward. I think doubling down will help children.

I understand that many of those who I disagree with say “we just need more time.” And I’m making a version of this argument here. So I’ve tried to lay out all my assumptions as clearly as possible, both for the sake of transparency, as well as with the hope that these assumptions can be corrected if they are wrong.

What do test scores tell us about schools?

Collin Hitt, Michael Q. McShane, and Patrick J. Wolf just published a report on the connection (or lack of connection) between test scores and long-term outcomes.

They looked at a bunch of school choices studies and tried to see if a school’s impact on student test scores was connected to its impact on student life outcomes.

Their conclusion: “at least for school choice programs, there is a weak relationship between impacts on test scores and later-life outcomes.”

Much of our K12 education policy is predicated on the idea that test scores are an important measure of school performance. If this is not true, behavior should change.

The Laura and John Arnold Foundation is currently funding more research on this question, and I’m eager to see what we find, especially with regards to income gains over time. This study was predicated on high school and college attainment being an indicator of long-term outcomes, but schooling isn’t always learning. So we should be careful judging school performance based on later school attainment, rather than income (or other measures).

But for now, here’s what’s on my mind after reading the study.

What do bad test score results tell us?

If I’m reading their report correctly (and I hope the authors correct me if I’m not), it seems rare that schools have a negative impact on test scores but a positive impact on long-term outcomes.

In the 126 study comparisons where test scores impacts were compared to high school or college outcomes, there were only 2 instances where a study found a significant negative impact on test scores and a significant positive impact on life outcomes.

It seems rare for a school to do really poorly on test scores but really great on life outcomes. I think the authors underemphasize this point in their paper.

What do mediocre test scores tell us?

It is much more common for schools to have a neutral (insignificant) impact on test scores but a significant positive impact on long-term outcomes. It looks like this occurred 32 times in the research review.

There may be a bunch of schools that don’t really impact test scores but are doing something that helps with long-term outcomes.

What do great test scores tell us?

There are no cases where a study found significantly positive test scores and significantly negative life outcomes.

So it seems rare for schools to jack up test scores but ruin kids lives. That’s good.

However, there were 17 instances of studies finding positive impact on test scores but neutral impacts on long-term outcomes.

So there seem to be some schools that are achieving good test results without translating these into great long-term outcomes.

How should this research affect regulation and philanthropy?

If these results hold, I think I will maintain my belief that we should replace schools with persistent very negative test scores. There appears to be little risk that these schools are really amazing schools. The negative test scores are a useful signal.

Yes, there might be other schools that are just as bad at life outcomes that are not closed because they achieve better test scores, but so long as we are closing schools that are not delivering great life outcomes, and opening schools that have a better chance of achieving great life outcomes, this seems like a worthwhile tradeoff.

But when it comes to expanding schools, if this research holds, I will rely less on positive test scores, and I think authorizers should do the same.

From an authorizer perspective, so long as a school does not have significantly negative test scores, perhaps the school should be able to expand so long as there is parent demand.

Philanthropy may also need to adjust by investing more heavily in school operators that show a positive impact on life outcomes (irregardless of test scores), and being willing to fund mediocre test score schools who either have high parent demand or who are using practices that are correlated with positive long-term outcomes (more research needed to determine what these might be).

I am very open to moving in this direction if research warrants it. The idea that it’s easier to tell a bad school than it is to identify a great school already matches my intuition, and deferring to parent judgment makes a lot of sense if we are not confident in our analysis of performance.

Did a federal grant to turnaround failing schools in New Orleans and Tennessee work?

Back when I worked at New Schools for New Orleans, we applied for a $30m federal grant to turnaround failing schools in New Orleans and scale the model to Tennessee.

CREDO just came out with a research study on our efforts. Their findings, and my analysis, are below.

The New Schools Were Much Better than the Ones They Replaced 

Here’s what CREDO found when they compared the schools we created to the schools we replaced:

In New Orleans, we replaced schools (“closing schools”) that were at 26th percentile in the state with new schools (“CRM schools”) that performed at the ~33rd percentile in the state at the end of the study.

In Tennessee, schools went from the ~17th percentile to the ~23rd percentile by the end of the study.

To quote the CREDO report: “the CRM schools in both New Orleans and Tennessee showed significantly higher academic growth compared to the Closing schools they replaced.”

Translated into days of learning, these are large effects: “Closing school students experience 63 fewer days of learning in reading and 86 fewer days of learning in math when compared to students in non-CRM schools… students in CRM schools make comparable academic growth to non-CRM students.”

The New Schools Performed About the Same as Other Schools in the City

When CREDO compared the new schools to other existing schools (rather than the failing schools they replaced), they found no statistically significant effects:

Screen Shot 2018-02-15 at 12.07.57 PM.png

In other words, the new schools that replaced the failing schools performed no better or worse than other existing schools in the city.

On one hand, this is disappointing. Our most ambitious targets included having the new schools be amongst the highest performing schools in the city.

On the other hand, this is still a major improvement: the new schools replaced failing schools and ended up achieving at the same level of most other schools in the city.

Building a System that Keeps Getting Better 

Replacing failing schools with new schools is a process, not a one-time intervention.

Ideally, a subset of the schools you created will do really well, and then, overtime, these schools will continue to grow. The ones that don’t do well will not be supported to do additional turnarounds.

Over the long-haul, gradually increasing the number and scale of high-quality school operators is more important than the average effect of the first wave of replacements.

Here’s what CREDO found across the new schools when they compared them to existing schools:

Screen Shot 2018-02-15 at 12.11.36 PM.pngIn New Orleans, 50% of the new schools had positive effects in both Math and Reading. This is really positive: half of our turnaround schools in New Orleans achieved significantly better results than existing schools across the city.

In Tennessee, only one school had positive effects in both Math and Reading, though a few other schools had positive effects in only reading.

This makes me optimistic that the school operator base in New Orleans will continue to have the capacity to replace more failing schools over time.

The early results in Tennessee are a bit more worrying on the operator quality front, and the next few years will be extremely important in ensuring that a healthy operator base emerges.

Lastly: CREDO found that replacing failing schools with fresh start schools (that opened one grade at a time) had a higher success rate than whole school turnarounds. My takeaway here is that you need a mature operator base to do a lot of whole school turnarounds, and no city had enough capacity to really do whole school at scale. In hindsight, we should have done more fresh starts and less whole school turnarounds.

Was the Effort a Success?

At the outset of the project, I remember debating with our research partners at CREDO about how to set-up the evaluation.

I argued that we should ultimately be judged on whether or not the new schools we created were better than the failing schools we replaced.

I didn’t think we should be primarily judged on whether or not the new schools were better than other existing schools that weren’t failing.

Yes, we did include language in the grant application that had goals of schools performing much better than existing schools. And as we executed the project we tried to pick school operators that we thought could deliver top tier results. Our highest aspirations weren’t met. This is disappointing, but it does not mean the project was a failure.

Rather, I consider the project to be a positive step forward in improving public education in these cities.

Making Things Better

The result of the project strikes at the heart of what’s so difficult about education reform: our aspirations for our most at-risk children are incredibly high, but making progress in creating better educational opportunities is very difficult.

In roughly a five year period, we replaced failing schools with new schools that were on average 7 percentile points higher in state performance, which translates to an extra 60-90 days of learning per year.

If the process of opening and replacement continues, what is a modest success right now may eventually become a great success.

I hope that this occurs and that New Orleans continues on its impressive track record of increasing student achievement. As a reminder, the federal grant was just one piece of an overall effort that has radically reduced failing schools in the city:

Screen Shot 2018-02-20 at 5.47.30 PM

SaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSaveSave

Does anyone know why Chicago children are getting smarter?

If you just read the headlines, you might suspect that Chicago’s public schools are in a terrible tailspin. Part of this is the noise of big city politics. Part of this stems from cloud of violence that hangs over the city.

But Chicago has improved on academic test scores more than most other cities in the country. Rather than one of the worst, Chicago is one of the best.

A recent report by Sean Reardon and Rebecca Hinze-Pifer found that between 2009 and 2014:

“This [student achievement] growth rate [of Chicago] is higher than 96% of all districts in the US. Among the 100 largest districts in the country, the average growth rate from third to eighth grade is 0.95 grade equivalents per year; Chicago has the highest growth rate between third and eighth grade of any large district in the United States.”

The authors admit that they don’t know why this occurred.

I can’t prove why Chicago kids are getting smarter, but I have a hypothesis.

What’s Been Going on in Chicago Public Schools? 

One way to try and solve the mystery of why Chicago children are getting smarter is to look at the district’s previous major initiatives.

As this report details, between roughy 1990 and 2010 there were three overarching eras of reform in Chicago: the decentralization era, the the accountability era, and the do a lot of things era.

The authors are very careful to not attribute a causal relationship between reform eras and outcomes. The reforms were messy and not rolled out in an experimental manner – so fair enough.

But in this post I’ll try and make my best guess on what was causal and what was not.

The Decentralization Era

The decentralization era was best known for the creation of Local School Councils. This reform gave local councils real control over decisions about how schools were run. The councils were made up of school leadership, parents, and community members.

The councils always seemed like a terrible idea to me. It’s basically taking all we know about charter schools (good central offices, scalable instructional programs, governance matters) and doing the exact opposite!

Not surprisingly, research on the reforms found that the councils had some positive effects on advantage communities, but were least likely to improve schools in low-income communites. Communities with low social capital didn’t gain a lot from ad-hoc and poorly constructed local boards.

I’m very skeptical that the decentralization era and school councils were the root cause of later gains.

The Accountability Era

In 1995 Mayor Daly put in Paul Valls as the superintendent (I later worked with Paul when he was the superintendent of the RSD in Louisiana). Vallas, who did not have deep instructional expertise, used test driven accountability to try and make things better.

New tests, promotional standards, and interventions for failing schools were all put in place.

The reforms had better impacts for low-performing schools; the researchers noted:

“This was the only era to show large improvements in the lowest- achieving schools. However, the patterns in test scores in the lowest-performing schools suggest that some of the improvements resulted from instruction that was aligned specifically to the high stakes tests.”

This matches other research on accountability reforms: you tend to see gains in the lowest preforming schools, but the high stakes can cause narrowing of the curriculum.

 

The Do a Lot of Things Era

Arne Duncan came in after Vallas, and he instituted a lot of reforms.

Arne launched 100 new schools, implemented internal district instructional and curricular reforms, overhauled school leadership pipelines, and placed a deep focus on on-time high school progression.

Perhaps the biggest initiative of this era was the Renaissance 2010 project, which launched about a 100 new district, charter, and contract schools between 2005 and 2010.

Unfortunately, no one has conducted a full evaluation of the program. Someone should do this!

Two interim research reports came out around 2010. One study, which only included a few years of data from the early Renaissance cohorts, found that the new schools performed about the same as the existing district schools. The other study was inconclusive.

Not much help from the research community.

A lot of work was also done on school leadership. The Chicago Public Education Fund, in partnership with the district, invested heavily in school leader development, placing bets on both district based and non-profit providers.

The lastest research I could find on these programs found that “results indicate that one-year learning gains in elementary and high schools led by Fund-supported principals were not different than those in other similar schools.”

Another major reform, another mediocre result.

All told, researchers found that this era produced more gains in high school than elementary schools, but wrote: “while the effects of the dominant policies of Eras 1 and 2 are largely understood, much research remains to be done to understand both the positive and problematic effects of the policies in Era 3.”

Not super helpful, especially since this is the era that preceded the large gains in test scores that occurred after 2009.

What About the Charter Sector? 

CREDO published a report on Chicago charters that covered test scores from 2010 to 2012, which is right in the middle of the period where Chicago saw a lot of gains.

The study found +.01 effects in reading and +.03 effects in math. These effects amount to about a month or so of extra learning per year, maybe a bit less. Given Chicago’s relatively small charter market share, and the modest size of these positive effects, it’s unlikely that charters themselves accounted for the 2009-2014 gains.

A more recent study, which just looked at charter high school performance from 2010-2013, found much larger effects: +.2 effects on ACT related tests and much higher college enrollment rates.

These are large effects, but they are for high school only. The study lauding Chicago’s gains only covered grades 3-8.

So WTF Happened in Chicago to Make Kids Smarter?

To summarize: Chicago improved its test scores more than any other big city in the country, and researchers really don’t know why.

So why are Chicago kids getting smarter?

Here’s my guess: competition and accountability lifted all boats.

When you put accountability in place (the Vallas era) and then launch a 100 new schools (the Duncan era) you get a city where school leaders know there are consequences for failure and the best of the new schools begin to raise the bar for what’s possible.

This theory helps explain why the Renaissance schools and charter effects were a bit muted. In the studies on these reforms, researchers compared the new schools to existing schools. So if the existing schools were improving due to increased competition, you would not see large relative effects for the new schools.

I can’t prove that accountability and competition caused the results, but in many sectors accountability and competition make everyone better. It also fits stories we’ve seen elsewhere. In place like Denver and Washington D.C. increased competition led to all boats rising in the public school system.

If you have a better theory, let me know.

What Should Chicago Do Now? 

Here’s another tough question: if it was accountability and competition that caused Chicago’s gains, how should this impact Chicago’s future strategy?

Since 2002 (while the district was getting much better!) Chicago enrollment plummeted from 440,000 students to 370,000 students.

This means that there are lot of under-enrolled schools in the district and the city might have to go through another round of painful closures.

This also it means it’s harder to push the very reform (opening new schools) that might have driven Chicago’s previous gains in achievement.

So what should the city do?

Reasonable people can surely disagree, but I would continue to create new schools, albeit in a different fashion.

First, I’d open new schools in the areas where population is increasing. Chicago is made up of a lot of neighborhoods, and not all neighborhoods are losing children.

Second, I would do some replacement work. Instead of closing all the under-enrolled schools, I’d try and select some neighborhoods where there’s enough child density that you could imagine families coming back to the public schools if there were better options. I’d launch replacement schools in these neighborhoods.

There are clear drawbacks to this strategy. Politically, it’s hard to justify opening schools when you’re in the midst of closures. Programmatically, it’s hard sell to get the operators of new schools to open up in neighbors with shrinking enrollment.

But I think it’s the best thing for children.

Lastly, I might also try and launch some diverse by design schools.

In a city as diverse as Chicago, it’s sad that it’s schools are so segregated.

The Last Word

Chicago’s Chief Education Officer, Janice Jackson, recently gave her take on why things are better.

Her list: pre-k, better professional development, better curriculum, competition from private and charter schools, and clear accountability standards.

In her own words:

“I believe the level of transparency we have provided around what a quality school is has been transformational in this district.”

Liberia is relinquishing. Is it working? 1st year results are in.

Screen Shot 2017-10-02 at 8.34.53 PM

Over the past few years, Liberia has embarked on an ambitious project to partner with non-governmental school networks.

Even more daring: Liberia’s political leadership is parterning with foreign school operators, some of which are for-profit.

Imagine for a second if, after Katrina, New Orleans political leaders had decided to partner with school operators from Singapore, Finland, and Shanghai.

In Liberia, numerous short-term and long-term risks abound – as do extremely high-potential upsides.

Partnership schools achieved .18 SD gains in one year 

Whatever one thinks of Liberia’s strategy, kudos to them for partnering with school networks in a manner that allowed for randomized control trials. Because schools were randomly selected for partnership, we can get a better understanding on whether or not the providers are delving a better education.

In aggregate, the first year effects were large: students in partnership schools scored 0.18 standard deviations higher in English and 0.18 standard deviations higher in mathematics than students in regular public schools. The authors note: “while starting from a very low level by international standards, this is the equivalent of 0.56 additional years of schooling for English and 0.66 additional years of schooling for math.”

Also, teachers are showing up more often: “teachers in partnership schools were 20 percentage points more likely to be in school during a random spot check (from a base of 40% in control schools).”

And teachers are teaching: “…16 percentage points more likely to be engaged in instruction during class time (from a base of 32% in control schools).”

Results varied by provider

The highest performing operators delivered ~.3 effects!: the Youth Movement for Collective Action (YMCA), Rising Academies, Bridge International Academies, and Street Child.

Second tier (very solid .15 effects): BRAC and More than Me.

Third tier (no effect): Omega and Stella Maris.

Screen Shot 2017-10-02 at 9.13.20 PM

Students receiving 2X learning time

This was incredible to me: “students in partnership schools spent twice as much time learning each week, when taking into account reduced absenteeism, increased time-on-task, and longer school days.”

Costs running a bit high (but to be expected in start-up)

The authors note that operators were spending more per-pupil than traditional schools; while this is a warning sign, I don’t read much into it now, as start-up efforts generally run higher and than smooth out. The exact same pattern happened in New Orleans.

What is the impact on traditional educators and schools?

As in the United States, non-governmental school operator growth impacts the traditional system, which has both programmatic and political consequences.

In an odd twist, the government contracts limited the class sizes of networks, which forced some operators to turn away students who then had to find other schools.

Operators also fired existing teachers, which presumably benefited children but caused adult hardship and risks political blowback.

I predict that these issues will only increase in salience. They require solutions that are both programmatic (government regulation of student equity issues) and political (ensuring that adult incumbents don’t derail positive efforts).

Teacher supply issues may get worse

The researchers note that to the extent that operators were able to recruit better teachers – and that the supply of teachers does not change – operators will be unable to scale and achieve the same effects.

In New Orleans we faced the same issue: we failed to grow high-quality teacher pipelines at the same pace we grew operators, and this caused operator growing pains midway through their scaling plans.

I hope Liberia gets ahead of this.

Is it worth it?

The perennial (and reasonable) question asked in such efforts is always: is it worth it? Is the disruption to families and educators worth the gain?

This question was asked a lot in New Orleans. I (as with the majority of New Orleanians) believe that it was worth it in New Orleans.

But I do think the Liberia case is more complicated, as it involves issues of national institutions and sovereignty.

There are numerous risks to outsourcing school operations to international organizations.

What if you end up in a conflict with the home nation(s) of large operators? What if these operators inculcate undesirable foreign values to your culture? What if the outsourcing of your educational operations slows down the overall maturation of your civil society?

These are hard questions.

My guess is that it is worth it, in that the gains of having a much better educated populace are worth the trade-offs of relying on foreign operators.

But I am not an expert in international development and I have not studied the issue enough to have strong opinions.

All that being said, all involved deserve our praise: the government is trying hard to serve their citizens, the school networks are serving students in extremely difficult situations, and the students themselves are getting smarter.

Here’s hoping the positive results continue.