Category Archives: Research

Liberia is relinquishing. Is it working? 1st year results are in.

Screen Shot 2017-10-02 at 8.34.53 PM

Over the past few years, Liberia has embarked on an ambitious project to partner with non-governmental school networks.

Even more daring: Liberia’s political leadership is parterning with foreign school operators, some of which are for-profit.

Imagine for a second if, after Katrina, New Orleans political leaders had decided to partner with school operators from Singapore, Finland, and Shanghai.

In Liberia, numerous short-term and long-term risks abound – as do extremely high-potential upsides.

Partnership schools achieved .18 SD gains in one year 

Whatever one thinks of Liberia’s strategy, kudos to them for partnering with school networks in a manner that allowed for randomized control trials. Because schools were randomly selected for partnership, we can get a better understanding on whether or not the providers are delving a better education.

In aggregate, the first year effects were large: students in partnership schools scored 0.18 standard deviations higher in English and 0.18 standard deviations higher in mathematics than students in regular public schools. The authors note: “while starting from a very low level by international standards, this is the equivalent of 0.56 additional years of schooling for English and 0.66 additional years of schooling for math.”

Also, teachers are showing up more often: “teachers in partnership schools were 20 percentage points more likely to be in school during a random spot check (from a base of 40% in control schools).”

And teachers are teaching: “…16 percentage points more likely to be engaged in instruction during class time (from a base of 32% in control schools).”

Results varied by provider

The highest performing operators delivered ~.3 effects!: the Youth Movement for Collective Action (YMCA), Rising Academies, Bridge International Academies, and Street Child.

Second tier (very solid .15 effects): BRAC and More than Me.

Third tier (no effect): Omega and Stella Maris.

Screen Shot 2017-10-02 at 9.13.20 PM

Students receiving 2X learning time

This was incredible to me: “students in partnership schools spent twice as much time learning each week, when taking into account reduced absenteeism, increased time-on-task, and longer school days.”

Costs running a bit high (but to be expected in start-up)

The authors note that operators were spending more per-pupil than traditional schools; while this is a warning sign, I don’t read much into it now, as start-up efforts generally run higher and than smooth out. The exact same pattern happened in New Orleans.

What is the impact on traditional educators and schools?

As in the United States, non-governmental school operator growth impacts the traditional system, which has both programmatic and political consequences.

In an odd twist, the government contracts limited the class sizes of networks, which forced some operators to turn away students who then had to find other schools.

Operators also fired existing teachers, which presumably benefited children but caused adult hardship and risks political blowback.

I predict that these issues will only increase in salience. They require solutions that are both programmatic (government regulation of student equity issues) and political (ensuring that adult incumbents don’t derail positive efforts).

Teacher supply issues may get worse

The researchers note that to the extent that operators were able to recruit better teachers – and that the supply of teachers does not change – operators will be unable to scale and achieve the same effects.

In New Orleans we faced the same issue: we failed to grow high-quality teacher pipelines at the same pace we grew operators, and this caused operator growing pains midway through their scaling plans.

I hope Liberia gets ahead of this.

Is it worth it?

The perennial (and reasonable) question asked in such efforts is always: is it worth it? Is the disruption to families and educators worth the gain?

This question was asked a lot in New Orleans. I (as with the majority of New Orleanians) believe that it was worth it in New Orleans.

But I do think the Liberia case is more complicated, as it involves issues of national institutions and sovereignty.

There are numerous risks to outsourcing school operations to international organizations.

What if you end up in a conflict with the home nation(s) of large operators? What if these operators inculcate undesirable foreign values to your culture? What if the outsourcing of your educational operations slows down the overall maturation of your civil society?

These are hard questions.

My guess is that it is worth it, in that the gains of having a much better educated populace are worth the trade-offs of relying on foreign operators.

But I am not an expert in international development and I have not studied the issue enough to have strong opinions.

All that being said, all involved deserve our praise: the government is trying hard to serve their citizens, the school networks are serving students in extremely difficult situations, and the students themselves are getting smarter.

Here’s hoping the positive results continue.

CREDO’s school closure research validates portfolio and golden tickets

 

CREDO just came out with a study on school closures. Matt Barnum gives a good write up in Chalkbeat (and continues to far surpass NYT and WAPO in his analysis of complicated research).

Achievement increases when you close low performing schools and students transfer to better schools 

Overall, the research increased my belief in the idea that great schools should expand and failing schools should be closed or transformed (the basis of the portfolio model).

My only reservation with this study is that it defined low-performing schools by absolute performance rather than growth in achievement; however, student achievement still grew when students moved from lower to higher absolute performing schools, so perhaps many of the low absolute schools were low growth schools as well.

The research adds to the body of evidence that shows: if you…

(1) Close lower performing schools;

(2) Increase the number of high-performing schools;

(3) Ensure students from the closed schools get into the higher-performing schools; then

(4) Educational opportunity will increase.

See below from CREDO on the student achievement effect of a student transferring from a closed school to a superior school.

Screen Shot 2017-08-27 at 9.12.18 AM

Screen Shot 2017-08-27 at 9.12.29 AM

One other thing: this type of analysis doesn’t capture the positive effect of the failing school no longer existing; a city implementing these strategies should see additional gains from no new students ever having to attend the failing school.

“Golden ticket” policies can ensure students attending closed schools get into better schools

Cities that use unified enrollment systems can easily guarantee that students leaving closed schools have access to better schools by a “golden ticket” policy. This policy gives students exiting closed schools first access to any open seats in high-quality schools in the city.

While I find the policy name “golden ticket” to be crass, it’s a policy that can do wonders for educational equity.

Too often students in failing schools are shuffled from one underperforming school to another; a golden ticket policy can prevent this.

Should we support increased school closures in majority white communities?

Numerous commentators pointed out that schools with +80% minority students were more likely to be closed than schools that had lower minority enrollment.

Most took this as a sign of inequitable treatment toward minority communities.

However, a growing body of research indicates that school closure increases educational opportunity so long as student have access to better schools. And although I wish this wasn’t the case, my hunch is that majority white communities likely have a higher concentration of better schools than your average minority community: this means that school closures are likely to be even more effective in majority white communities.

My guess is a lack of closures in majority white communities is leading to reduced educational opportunity.

School closure is incredibly hard

While I believe that thoughtfully implemented school closure policies will benefit children, I know that closing schools is hard for students, families, educators, and politicians.

But sometimes doing the hard thing can help students.

And a growing body of evidence is pointing to the idea that a well implemented school closures is one of those hard things that can ultimately make things better for students and communities.

__

Lastly: thinking of everyone in Houston right now. Really hoping that friends, colleagues, and everyone else in city are ok.

SaveSave

SaveSave

The KIPP pre-K study doesn’t tell us if KIPP pre-k works

Mathematica just published a study on KIPP pre-k. The Laura and John Arnold Foundation (where I work) funded the study.

The study was well designed and asked a very important question: does high-quality charter pre-k provide benefits above and beyond simply attending a  high-quality charter elementary school?

The policy implications of this question are important: if high-quality charter pre-k adds to the achievement of students who attend high-performing elementary charter schools, policies that support both early child funding and charter school growth could accelerate achievement gains.

However, if attending a high-quality charter pre-k has no effect beyond the impact of simply attending a high-quality charter elementary school, then our limited public resources may be best spent on expanding high-quality charter elementary schools and not devoting as much resources to pre-k.

In other words, it’s really important to understand if:

KIPP pre-K + KIPP elementary > KIPP elementary

Unfortunately, due to small sample sizes as well as sample attrition, the researchers were not able to answer this question.

While the effects of KIPP pre-K + elementary were larger than the effects of KIPP elementary, these results were not significant.

The researchers write:

In reading, the magnitude of the positive impact was larger for the pre-K cohort than the kindergarten cohort on both the Letter-Word and Passage Comprehension tests administered in grade 2 (by 0.20 and 0.06 standard deviation units, respectively). Neither of these differences is statistically significant; however, the study did not have sufficient power to detect differences of this magnitude. Thus, they may be suggestive of some additional benefit in reading resulting from an offer of admission to KIPP pre-K, above and beyond the impact of an offer to KIPP in kindergarten.

In math, however, the impacts for both samples are identical, suggesting no additional benefit of KIPP pre-K beyond the impact of a KIPP elementary school.

Below you can see the estimated (non-significant) impacts:

 

Screen Shot 2017-08-23 at 12.23.34 PM.png

So, for now, we still don’t know if KIPP pre-k adds any value over KIPP elementary school.

This is no knock on KIPP or the researchers, it’s just the consequence of the limitations of working with small sample sizes that are subject to attrition.

That being said, I do think some of my friends are playing a little fast and loose in over interpreting this study, and I think we all need to be honest about the fact that we don’t yet know if KIPP pre-k “works.”

Phase II of the charter and choice research agenda is extremely important

In a recent post, I summarized much of the research on charter school and high choice cities.

The short of it: there’s a strong body of evidence that urban charter schools outperform traditional schools, and a nascent body of evidence that thoughtfully implemented high choice cities can deliver strong achievement gains.

It’s worth taking a moment to celebrate this: the research on urban charter schools is an impressive body of work built on twenty years of studying the incredible efforts of educational entrepreneurs. It’s wonderful to see these schools working.

___

But as charters scale, additional questions need to be studied.

More specifically:

  1. Do urban charter school students achieve better long-term life outcomes?
  2. What happens to traditional schools when charter schools expand?
  3. What happens to students when under-enrolled traditional schools eventually close?
  4. What happens to cities that transition to majority charter systems?

A few recent studies have shown that charter school students achieve better long-term life outcomesthat charter school expansion can improve traditional schools; that student attending schools that are closed actually benefit from the closure so long as good new schools are being continually opened; and, as noted in the previous post, that majority charter cities can deliver strong achievement gains.

But there are also studies that indicate the opposite. Charter students in Texas did not have greatly improved life outcomes; students in Baton Rouge did not increase achievement after their schools were closed; and cities such as Detroit continue to struggle despite high charter penetration.

___

Personally, I most care about questions (1) and (4). If charter schools end up delivering better life outcomes, and majority charter cities work, I’d be willing to accept some interim negative effects of charter expansion and school closures. But I understand that the latter effects could cause hardship for families, and I’m heartened to see early examples that show that charter expansion can be coupled with gains for all students.

So here’s to holding out for the best scenario: I truly hope that phase II of the research agenda shows that charters increase life outcomes, that charter school expansion and failing school closure benefits all students, and that majority charter systems deliver benefits for all students.

Initial research indicates that this might be possible, which is really exciting.

Does it matter if charter schools make traditional schools better?

A new study found that traditional schools in New York City improved when charter schools opened up in the near vicinity. See this Chalkbeat article for coverage.

The effects were small for neighborhood impact (traditional schools improved +.02 SDs) and larger for co-locations (traditional schools improved +.06-.09 SDs).

What is the value of charter schools?

Charter schools could help students in two ways: (1) directly serving students and (2) increasing the performance of other schools.

I don’t believe that #2 must be true in order to support charter schools. The very fact that charter schools help the students they serve could be enough to warrant support.

An even more complicated question: what if charter schools increased the performance of the students they served but decreased the performance of traditional schools?

If, like me, you believed non-profit operation of schools should overtime supplant direct government operation of schools, then you still might support the expansion of charter schools. The negative impact of charter growth on traditional schools, while painful, could be a price worth paying if more children were benefitted by the new system over the long-haul.

Be careful how you justify your opinions 

The reason I make this point is that I don’t think we should tie the value of charter schools to their impact on nearby traditional schools.

While this impact is something we should understand – and if in some instances there are negative impacts we should we work to reduce them – ultimately, the fact that their might be a better way to do public education should be our Northstar.

Building the new system is more important to children than protecting the old system.

But hopefully we continue to find more evidence of win / win

That being said, this study is good news: it’s pretty amazing that New York City has grown a high-quality charter sector and that this sector has helped improved traditional schools.

Hopefully future studies will show the same.

But they might not. And expanding great public charter schools might still be the best thing for children.

So we should be cautious in justifying the existence of charter schools in how they impact traditional schools.

Ultimately, their highest and best use is likely to be in the creation of a better way to do public schooling at scale.

Personalized learning is a transformative idea without a transformative technology

Screen Shot 2017-07-11 at 4.59.53 PM.png

I just got back from vacation, which was a great time to read the The Three Body Problem science fiction trilogy, a wonderful series that revolves around the protoganist using first principles thinking to negotiate with an alien species.

Upon return, I read this Rand report on personalized learning, which was funded by the Gates foundation. The report covers a small set of schools in the early years of implementation, so best not to draw too firm of conclusions.

The report found:

  • Charters that adopted personalized learning strategies saw a +.1 effect in math and no statistically significant in reading.
  • District schools (very small N) saw no achievement gains.
  • Charter schools implemented personalized learning strategies with more operational fidelity.

Perhaps most interestingly, the authors noted:

In this theoretical conception, schools that are high implementers of PL [personalized learning]  approaches would look very different from more traditional schools. In practice, although there were some differences between the NGLC schools and the national sample, we found that schools in our study were implementing PL approaches to a varying degree, with none of the schools looking as radically different from traditional schools as theory might predict.

So in this sample, charters outperform traditional schools (thought by a lesser margin than urban charters as a whole outperform traditional schools); charters execute better; and the schools themselves don’t look radically different than traditional schools.

Hence the title of this post: personalized learning is a transformative idea without a transformative technology.

Without a technological breakthrough, the current personalized learning efforts will, at best, lead to modest improvements on the execution of common place ideas (using data to drive instruction, executing leveled small group instruction, investing children in goals, etc.). School will look the same and be a little more effective and pleasant for all involved.

This is fine and the world is in many ways built on modest improvements.

But for personalized learning to live up to its hype (as well as to its philanthropic investment), it will need a technological breakthrough.

Instructional platforms might be the first breakthrough, but even here I think the primary effects will be more around scaling great school models and content rather than deep personalization.

The crux of the issue is this: computers are simply not as good as humans in coaching students through instructional problems.

Your average person off the street remains a more effective grade school tutor than the most powerful computer in the world.

Until this changes, personalized learning will never realize its promise. The problem is one of technology, not practice.

My response to Freddie deBoer

Freddie deBoer recently commented on my post on educational productivity. Freddie is a rising star in the blogosphere, and his concerns mirror the concerns of many others, so I thought responding to his comment could be of use.

Below, I respond to his critiques. With Freddie’s permission, I’ve also pasted in his full comment at the end of the post. It might be helpful to start there.

I. Do Charters Outperform Traditional Schools? 

Freddie begins:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools.

This is true when it comes to all students. But is not true when it comes to disadvantaged students, especially those served by urban charter schools.

CREDO’s urban charter study found the following effects for urban charters:

Screen Shot 2015-03-18 at 7.39.20 PM

The study also found positive charter effects for disadvantaged populations:

Screen Shot 2017-06-15 at 8.51.23 AM

CREDO’s national study found similar positive effects for disadvantaged populations.

CREDO is the best national quasi-experimental data source we have, and its methodology holds up well in comparisons with experimental data.

To the extent you’re suspicious of CREDO or of quasi-experimental design, Rand also did a national study on charters that looked exclusively at experimental studies. The authors found:

“Consistent with many previous studies that have focused on broad sets of charter schools, we found no evidence that, on average, attending charter schools had a positive impact on student achievement. The estimated impact of attending the average charter school in the study was negative but not statistically signicant after adjusting for the multiple hypotheses tested. However, the average impact of attending charter schools in large urban areas or those serving lower achieving or more disadvantaged students was large and positive.”

In a NBER working paper, Dennis Epple, Richard Romano, and Ron Zimmer offer their summary of charter RCT research:

“These studies have been much more supportive of charter schools with nearly all of these studies finding positive effects—in some cases, quite large effects (Hoxby and Rockoff, 2004; Hoxby, Kang, & Murarka, 2009; Abdulkadiroglu, et al., 2010; Curto and Fryer, 2011; Tuttle, et al., 2013; Wong, et al., 2014)—with only one finding no effect, a study by Mathematica of charter middle schools (Gleason, et al., 2010).”

For what it’s worth, my reading of the Mathematica study (which the above authors say is the only finding with no effect), is that it does find positive math impacts in for disadvantaged students, see pages 70-71,78 in this report. 

Apple, Romano, and Zimmer also provide a good analysis of the pros and cons of quasi-expermintal studies and experimental studies, as well as the trickiness of solving for selection effects.

While none of the evidence is perfect, I think it’s very reasonable to hold the belief that charters are serving disadvantaged students in urban areas better than traditional public schools.

At the very least, I would not call this belief “a basic failed assumption.”

II. Can We Replicate the Success of New Orleans?

Freddie writes:

Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities. So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

I don’t know if the New Orleans efforts can be scaled, but I don’t think the idea is absurd.

Rather, I think we should see if the structural reforms of New Orleans can be applied with success in additional cities.

Very specifically, I think it would be great to get 8-10 cities where:

(1) A majority of the schools are non-profit managed.

(2) A unified enrollment system allows families to easily choose from a variety of schools.

(3) A unified accountability system provides parents information about school quality and leads to governance change in the lowest performing schools.

I view these as the core tenets of the New Orleans model.

Big picture, I don’t understand why we’d dismiss the incredible achievement gains in New Orleans rather than try to learn from them and see if they can work elsewhere.

As a reminder, the New Orleans achievement gains were very large:

Screen Shot 2015-08-04 at 7.03.28 AM

III. Can We Replicate the Success of High-Performing Charter Management Organizations? 

Freddie writes:

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs.
Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent. These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed.

I agree with Freddie that Success Academy might not be scalable outside of New York City (or another major urban area like Chicago or Los Angeles). Freddie points out real potential limitations to their model.

That being said, if Success Academy could provide great educational opportunities to 300,000-500,000 students, I’d be hesitant to dismiss their impact simply because they won’t reach every child in the country.

Success Academy aside, I believe that Freddie is incorrect that successful charter organizations can’t scale.

Take KIPP, which now serves nearly 100,000 students (including schools in the Arkansas Delta!).

Mathematica just completed a rigorous analysis of KIPP and found that “KIPP schools have positive, statistically significant, and educationally meaningful impacts on student achievement, particularly at the elementary and middle school levels.”

See here for their middle school effects:

Screen Shot 2017-06-15 at 10.37.30 AM

And KIPP is not an outlier.

The Charter School Growth Fund, which supports the growth of charter organizations across the country, recently had its portfolio analyzed by CREDO, who found strong effects:

Screen Shot 2017-06-12 at 8.07.41 PM

Charter School Growth’s portfolio serves about 300,000 students and could feasibly scale to over a million students in the coming decade.

Admittedly, charters have not yet, at scale, achieved significant student achievement gains with suburban and rural populations. Maybe one day they will. Charter organizations such as IDEA Public Schools have achieved +.1 effects in places like the Rio Grande Valley.

But even if charter end up not being the right solution for rural areas, why not support the best charters to provide great educational opportunities to millions of disadvantaged students in urban areas?

IV. Are Emerging Choice Markets Working?

Freddie writes:

Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely.

I was surprised Freddie made this argument. In CREDO’s study of urban charters, all four of these cities achieved positive charter effects. Admittedly, Detroit’s charter sector is not a shining star, but it is still outperforming the traditional sector.

Here are the charter effects for these cities in Math:

 

And here are the effects for ELA:

 

Again, if you don’t trust CREDO, independent researchers also found that Washington D.C. made meaningful achievement gains over the past decade. This report from the Urban Institute came to similar conclusions. Other experimental research in choice markets such as Denver also finds strong effects.

I really don’t understand the claim that these markets have failed completely. Newark, Nashville, and Washington D.C. are three of the stronger charter school markets in the country.

V. Will Positive Test Score Results Lead to Good Life Outcomes?

Freddie writes:

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.

Freddie raises fair concerns about the logic jump that increasing test scores will lead to positive life outcomes.

Raj Chetty has done the deepest work on the connection between test score gains and life outcomes (he found a positive link), but I don’t view this work as conclusive. Similar studies that are focused on long-term outcomes for charter students have found both positive and insignificant life effects.

I view this as an area where we don’t have enough evidence to make strong claims.

If I had to guess, I would say that the early charter movement focused too narrowly on test scores and is now evolving to focus much more on life outcomes, and that, over time, we will see the same success in life outcomes as we have seen in test scores.

All that being said, we may find out that improving student outcomes in K12 just does not translate into long-term life gains.

With regards to whether I’m a serious broker or an ideologue, it’s worth noting that I publicly stated my fears on this issue in a previous blog post entitled: The Current Brutal Reality of Education Reform and Wage Growth.

In the post, I reviewed the disappointing data Fryer found on life outcomes of charter students and ended with the following:

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

I think about this a lot. It’s also in part why I write this blog. Reviewing rigorous research and putting my ideas out there for public critique are attempts to make sure I’m not deluding myself into holding false beliefs.

VI. Conclusion

I believe the available experimental and quasi-experimental data support the belief the charters are doing a good job raising test scores for disadvantaged students.

I believe the work of organizations such as KIPP and Charter School Growth Found support the belief that high-quality charter schools could grow to serve millions of students.

I believe the early results of choice markets in places like New Orleans and Washington D.C. provide evidence that well regulated city choice markets may lead to better results at scale.

I believe we do not currently know if test score gains will translate into positive life outcomes.

Ultimately, those of us working in the educator sector work in a field where very few interventions work. In this sense, I appreciate Freddie’s general skepticism. I think reformers have too often promised too much.

But I think there’s a chance that urban charters and city choice systems can lead to better educational opportunities for millions of disadvantaged students.

So that’s why I do what I do, operating as much as possible with the full awareness that I might be wrong.

____

Freddie’s original comment:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools. Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities.

So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs. Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent.

These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed. Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely. Which do you think will be more likely to be scaled by hundreds of thousands of schools and millions of teachers?

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.