Category Archives: Research

Phase II of the charter and choice research agenda is extremely important

In a recent post, I summarized much of the research on charter school and high choice cities.

The short of it: there’s a strong body of evidence that urban charter schools outperform traditional schools, and a nascent body of evidence that thoughtfully implemented high choice cities can deliver strong achievement gains.

It’s worth taking a moment to celebrate this: the research on urban charter schools is an impressive body of work built on twenty years of studying the incredible efforts of educational entrepreneurs. It’s wonderful to see these schools working.

___

But as charters scale, additional questions need to be studied.

More specifically:

  1. Do urban charter school students achieve better long-term life outcomes?
  2. What happens to traditional schools when charter schools expand?
  3. What happens to students when under-enrolled traditional schools eventually close?
  4. What happens to cities that transition to majority charter systems?

A few recent studies have shown that charter school students achieve better long-term life outcomesthat charter school expansion can improve traditional schools; that student attending schools that are closed actually benefit from the closure so long as good new schools are being continually opened; and, as noted in the previous post, that majority charter cities can deliver strong achievement gains.

But there are also studies that indicate the opposite. Charter students in Texas did not have greatly improved life outcomes; students in Baton Rouge did not increase achievement after their schools were closed; and cities such as Detroit continue to struggle despite high charter penetration.

___

Personally, I most care about questions (1) and (4). If charter schools end up delivering better life outcomes, and majority charter cities work, I’d be willing to accept some interim negative effects of charter expansion and school closures. But I understand that the latter effects could cause hardship for families, and I’m heartened to see early examples that show that charter expansion can be coupled with gains for all students.

So here’s to holding out for the best scenario: I truly hope that phase II of the research agenda shows that charters increase life outcomes, that charter school expansion and failing school closure benefits all students, and that majority charter systems deliver benefits for all students.

Initial research indicates that this might be possible, which is really exciting.

Does it matter if charter schools make traditional schools better?

A new study found that traditional schools in New York City improved when charter schools opened up in the near vicinity. See this Chalkbeat article for coverage.

The effects were small for neighborhood impact (traditional schools improved +.02 SDs) and larger for co-locations (traditional schools improved +.06-.09 SDs).

What is the value of charter schools?

Charter schools could help students in two ways: (1) directly serving students and (2) increasing the performance of other schools.

I don’t believe that #2 must be true in order to support charter schools. The very fact that charter schools help the students they serve could be enough to warrant support.

An even more complicated question: what if charter schools increased the performance of the students they served but decreased the performance of traditional schools?

If, like me, you believed non-profit operation of schools should overtime supplant direct government operation of schools, then you still might support the expansion of charter schools. The negative impact of charter growth on traditional schools, while painful, could be a price worth paying if more children were benefitted by the new system over the long-haul.

Be careful how you justify your opinions 

The reason I make this point is that I don’t think we should tie the value of charter schools to their impact on nearby traditional schools.

While this impact is something we should understand – and if in some instances there are negative impacts we should we work to reduce them – ultimately, the fact that their might be a better way to do public education should be our Northstar.

Building the new system is more important to children than protecting the old system.

But hopefully we continue to find more evidence of win / win

That being said, this study is good news: it’s pretty amazing that New York City has grown a high-quality charter sector and that this sector has helped improved traditional schools.

Hopefully future studies will show the same.

But they might not. And expanding great public charter schools might still be the best thing for children.

So we should be cautious in justifying the existence of charter schools in how they impact traditional schools.

Ultimately, their highest and best use is likely to be in the creation of a better way to do public schooling at scale.

Personalized learning is a transformative idea without a transformative technology

Screen Shot 2017-07-11 at 4.59.53 PM.png

I just got back from vacation, which was a great time to read the The Three Body Problem science fiction trilogy, a wonderful series that revolves around the protoganist using first principles thinking to negotiate with an alien species.

Upon return, I read this Rand report on personalized learning, which was funded by the Gates foundation. The report covers a small set of schools in the early years of implementation, so best not to draw too firm of conclusions.

The report found:

  • Charters that adopted personalized learning strategies saw a +.1 effect in math and no statistically significant in reading.
  • District schools (very small N) saw no achievement gains.
  • Charter schools implemented personalized learning strategies with more operational fidelity.

Perhaps most interestingly, the authors noted:

In this theoretical conception, schools that are high implementers of PL [personalized learning]  approaches would look very different from more traditional schools. In practice, although there were some differences between the NGLC schools and the national sample, we found that schools in our study were implementing PL approaches to a varying degree, with none of the schools looking as radically different from traditional schools as theory might predict.

So in this sample, charters outperform traditional schools (thought by a lesser margin than urban charters as a whole outperform traditional schools); charters execute better; and the schools themselves don’t look radically different than traditional schools.

Hence the title of this post: personalized learning is a transformative idea without a transformative technology.

Without a technological breakthrough, the current personalized learning efforts will, at best, lead to modest improvements on the execution of common place ideas (using data to drive instruction, executing leveled small group instruction, investing children in goals, etc.). School will look the same and be a little more effective and pleasant for all involved.

This is fine and the world is in many ways built on modest improvements.

But for personalized learning to live up to its hype (as well as to its philanthropic investment), it will need a technological breakthrough.

Instructional platforms might be the first breakthrough, but even here I think the primary effects will be more around scaling great school models and content rather than deep personalization.

The crux of the issue is this: computers are simply not as good as humans in coaching students through instructional problems.

Your average person off the street remains a more effective grade school tutor than the most powerful computer in the world.

Until this changes, personalized learning will never realize its promise. The problem is one of technology, not practice.

My response to Freddie deBoer

Freddie deBoer recently commented on my post on educational productivity. Freddie is a rising star in the blogosphere, and his concerns mirror the concerns of many others, so I thought responding to his comment could be of use.

Below, I respond to his critiques. With Freddie’s permission, I’ve also pasted in his full comment at the end of the post. It might be helpful to start there.

I. Do Charters Outperform Traditional Schools? 

Freddie begins:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools.

This is true when it comes to all students. But is not true when it comes to disadvantaged students, especially those served by urban charter schools.

CREDO’s urban charter study found the following effects for urban charters:

Screen Shot 2015-03-18 at 7.39.20 PM

The study also found positive charter effects for disadvantaged populations:

Screen Shot 2017-06-15 at 8.51.23 AM

CREDO’s national study found similar positive effects for disadvantaged populations.

CREDO is the best national quasi-experimental data source we have, and its methodology holds up well in comparisons with experimental data.

To the extent you’re suspicious of CREDO or of quasi-experimental design, Rand also did a national study on charters that looked exclusively at experimental studies. The authors found:

“Consistent with many previous studies that have focused on broad sets of charter schools, we found no evidence that, on average, attending charter schools had a positive impact on student achievement. The estimated impact of attending the average charter school in the study was negative but not statistically signicant after adjusting for the multiple hypotheses tested. However, the average impact of attending charter schools in large urban areas or those serving lower achieving or more disadvantaged students was large and positive.”

In a NBER working paper, Dennis Epple, Richard Romano, and Ron Zimmer offer their summary of charter RCT research:

“These studies have been much more supportive of charter schools with nearly all of these studies finding positive effects—in some cases, quite large effects (Hoxby and Rockoff, 2004; Hoxby, Kang, & Murarka, 2009; Abdulkadiroglu, et al., 2010; Curto and Fryer, 2011; Tuttle, et al., 2013; Wong, et al., 2014)—with only one finding no effect, a study by Mathematica of charter middle schools (Gleason, et al., 2010).”

For what it’s worth, my reading of the Mathematica study (which the above authors say is the only finding with no effect), is that it does find positive math impacts in for disadvantaged students, see pages 70-71,78 in this report. 

Apple, Romano, and Zimmer also provide a good analysis of the pros and cons of quasi-expermintal studies and experimental studies, as well as the trickiness of solving for selection effects.

While none of the evidence is perfect, I think it’s very reasonable to hold the belief that charters are serving disadvantaged students in urban areas better than traditional public schools.

At the very least, I would not call this belief “a basic failed assumption.”

II. Can We Replicate the Success of New Orleans?

Freddie writes:

Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities. So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

I don’t know if the New Orleans efforts can be scaled, but I don’t think the idea is absurd.

Rather, I think we should see if the structural reforms of New Orleans can be applied with success in additional cities.

Very specifically, I think it would be great to get 8-10 cities where:

(1) A majority of the schools are non-profit managed.

(2) A unified enrollment system allows families to easily choose from a variety of schools.

(3) A unified accountability system provides parents information about school quality and leads to governance change in the lowest performing schools.

I view these as the core tenets of the New Orleans model.

Big picture, I don’t understand why we’d dismiss the incredible achievement gains in New Orleans rather than try to learn from them and see if they can work elsewhere.

As a reminder, the New Orleans achievement gains were very large:

Screen Shot 2015-08-04 at 7.03.28 AM

III. Can We Replicate the Success of High-Performing Charter Management Organizations? 

Freddie writes:

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs.
Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent. These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed.

I agree with Freddie that Success Academy might not be scalable outside of New York City (or another major urban area like Chicago or Los Angeles). Freddie points out real potential limitations to their model.

That being said, if Success Academy could provide great educational opportunities to 300,000-500,000 students, I’d be hesitant to dismiss their impact simply because they won’t reach every child in the country.

Success Academy aside, I believe that Freddie is incorrect that successful charter organizations can’t scale.

Take KIPP, which now serves nearly 100,000 students (including schools in the Arkansas Delta!).

Mathematica just completed a rigorous analysis of KIPP and found that “KIPP schools have positive, statistically significant, and educationally meaningful impacts on student achievement, particularly at the elementary and middle school levels.”

See here for their middle school effects:

Screen Shot 2017-06-15 at 10.37.30 AM

And KIPP is not an outlier.

The Charter School Growth Fund, which supports the growth of charter organizations across the country, recently had its portfolio analyzed by CREDO, who found strong effects:

Screen Shot 2017-06-12 at 8.07.41 PM

Charter School Growth’s portfolio serves about 300,000 students and could feasibly scale to over a million students in the coming decade.

Admittedly, charters have not yet, at scale, achieved significant student achievement gains with suburban and rural populations. Maybe one day they will. Charter organizations such as IDEA Public Schools have achieved +.1 effects in places like the Rio Grande Valley.

But even if charter end up not being the right solution for rural areas, why not support the best charters to provide great educational opportunities to millions of disadvantaged students in urban areas?

IV. Are Emerging Choice Markets Working?

Freddie writes:

Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely.

I was surprised Freddie made this argument. In CREDO’s study of urban charters, all four of these cities achieved positive charter effects. Admittedly, Detroit’s charter sector is not a shining star, but it is still outperforming the traditional sector.

Here are the charter effects for these cities in Math:

 

And here are the effects for ELA:

 

Again, if you don’t trust CREDO, independent researchers also found that Washington D.C. made meaningful achievement gains over the past decade. This report from the Urban Institute came to similar conclusions. Other experimental research in choice markets such as Denver also finds strong effects.

I really don’t understand the claim that these markets have failed completely. Newark, Nashville, and Washington D.C. are three of the stronger charter school markets in the country.

V. Will Positive Test Score Results Lead to Good Life Outcomes?

Freddie writes:

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.

Freddie raises fair concerns about the logic jump that increasing test scores will lead to positive life outcomes.

Raj Chetty has done the deepest work on the connection between test score gains and life outcomes (he found a positive link), but I don’t view this work as conclusive. Similar studies that are focused on long-term outcomes for charter students have found both positive and insignificant life effects.

I view this as an area where we don’t have enough evidence to make strong claims.

If I had to guess, I would say that the early charter movement focused too narrowly on test scores and is now evolving to focus much more on life outcomes, and that, over time, we will see the same success in life outcomes as we have seen in test scores.

All that being said, we may find out that improving student outcomes in K12 just does not translate into long-term life gains.

With regards to whether I’m a serious broker or an ideologue, it’s worth noting that I publicly stated my fears on this issue in a previous blog post entitled: The Current Brutal Reality of Education Reform and Wage Growth.

In the post, I reviewed the disappointing data Fryer found on life outcomes of charter students and ended with the following:

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

I think about this a lot. It’s also in part why I write this blog. Reviewing rigorous research and putting my ideas out there for public critique are attempts to make sure I’m not deluding myself into holding false beliefs.

VI. Conclusion

I believe the available experimental and quasi-experimental data support the belief the charters are doing a good job raising test scores for disadvantaged students.

I believe the work of organizations such as KIPP and Charter School Growth Found support the belief that high-quality charter schools could grow to serve millions of students.

I believe the early results of choice markets in places like New Orleans and Washington D.C. provide evidence that well regulated city choice markets may lead to better results at scale.

I believe we do not currently know if test score gains will translate into positive life outcomes.

Ultimately, those of us working in the educator sector work in a field where very few interventions work. In this sense, I appreciate Freddie’s general skepticism. I think reformers have too often promised too much.

But I think there’s a chance that urban charters and city choice systems can lead to better educational opportunities for millions of disadvantaged students.

So that’s why I do what I do, operating as much as possible with the full awareness that I might be wrong.

____

Freddie’s original comment:

You’re making the most basic failed assumptions possible in this post. At scale, charters are not significantly different from public schools. Charters that show these gains are idiosyncratic examples that receive the benefit of unusual structural advantages and advantages of massive effort, attention, and time from deep-pocketed entities.

So you get examples like New Orleans, where an army of do-gooders descended and the entire civic infrastructure was remade top-to-bottom and suggest that can be meaningfully scaled, which is absurd.

Or Success Academy, where teachers churn in and out of the system at something like twice the (already sky-high) attrition rate for teachers, and can be replaced by a never-ending stream of people with Ivy League degrees looking for their first NYC jobs who are willing to work under intensely unhappy working conditions for relatively low pay, and then after a few years move on to more remunerative jobs. Try that in the Ozark mountains or the Mississippi Delta and see if you can attract that kind of talent.

These systems also tend to be filled with hidden selection bias, as was found by Reuters in a huge investigation of the many ways charters cook the books to only admit the students most likely to succeed. Meanwhile in places like Detroit, Nashville, Newark, and Washington DC choice programs have failed completely. Which do you think will be more likely to be scaled by hundreds of thousands of schools and millions of teachers?

“Charter” simply is not a condition that can be scaled; it’s not really a consistent condition at all. The fact that you wave your hand and blithely assume that what worked in the totally idiosyncratic case of New Orleans – presuming there’s no fraud going on and that the test score advantages won’t degrade over time, and that we see actual differences in college-level persistence and success, a big question – shows that you’re not a serious broker. You’re an ideologue.

The Hondas are coming

CREDO just released its new research report on the performance of charter school management organizations (CMOs).

I continue to be grateful for CREDO’s efforts. In for-profit industries, the market creates demand for this type of sector specific research. In the social sector, we tend to be more reliant on academics and philanthropists; thankfully, CREDO continues to drum up the necessary support to produce this type of analysis.

Positive, Modest  Effect Sizes Everywhere You Look

CMOs are delivering solid effects with most, but not all students. It is easy to brush off these effects as “smallish.” But one could say the same of many groundbreaking innovations that make the world a better place.

We should be optimistic about the fact that CMOs tend to deliver Honda-like performance improvements: they are better than existing model, and their value increases the longer you use them.

Overall, CMOs are delivering +.03 SD effects over three years in both reading and math. These gains are driven by the fact that students benefit from CMOs the longer they stay in them:

Screen Shot 2017-06-12 at 7.45.24 PMScreen Shot 2017-06-12 at 7.45.12 PM

Yes, some individual CMOs operate at the frontier of innovation: they develop new school models or operate with world class execution. And while we should praise these efforts, we should not ignore the more modest improvements that are being delivered at scale by the CMO sector itself.

Give or take, CMOs serve around a million children, and many of these children are getting a better education than they would receive otherwise.

For Disadvantaged Students, the Benefits of CMOs are Twice as Good 

As with Hondas, the benefits of CMOs generally accrue to those who need them the most.

The charts below are a little tricky to interpret, but they show that African-American students see roughly double the positive effects (+.06 SD instead of +.03SD) when they enroll in CMOs (the difference between the two bars equals the marginal CMO effect).

For minority students in poverty, the effects were even bigger, equating to around a .1 effect for Hispanic and African-American students in poverty.

The only disadvantaged population to see modest negative effects was students with special needs. The sector needs to get better here.

A Good Investor to Grow the Sector

Many philanthropists do not give directly to CMOs. Instead, they give to the Charter School Growth Fund, whose management team and board then make decisions on which individual CMOs to invest in.

Over the decade, the portfolio of the Charter School Growth Fund has significantly increased in size. To date, they have also managed to maintain quality.

Screen Shot 2017-06-12 at 8.07.41 PM

Schools in the Charter School Growth Fund portfolio are delivering much greater effects than the CMO community as a whole.

Almost Nothing Works, So Nurture the Efforts that Do

Very few education interventions achieve positive results.

CMOs achieve positive results. And these results continue to hold as they scale.

Under less than ideal political conditions, and sometimes with little public support, these organizations are doing a lot for disadvantaged youth.

If we continue to support their development, our nation could be much better off.

Roland Fryer and the Root Cause of Good Management

Screen Shot 2017-05-30 at 3.44.45 PM

Roland Fryer is one of the top education researchers in the country. His research is always thought provoking and whenever we talk I learn something.

If there’s one area Roland and I may disagree on, it’s the potential for school districts to sustainably adopt the best practices of charter schools (which Roland has been instrumental in helping us understand).

This issue is of course wrapped up in the bigger question: will the greatest value of charter schools be the birth of  innovative practices or the scaling of a better governance model?

I. Roland’s New Research: MGMT Matters

Roland just came out with a fascinating study on the importance of effective principal management.

The experimental research project was set in Houston and provided principal management training (much of it borrowed from Paul Bambrick-Santoyo of Uncommon Schools) to a treatment group of school district principals.

The researchers found:

Overall, the estimates suggest that management training was effective in year one – increasing efficiency approximately 7% — but produced precisely estimated zeros in year two. Pooling the two years produces marginally significant results that fall on the other side of significance with more conservative standard errors. Management training tends to be more effective with more flexible, stable and higher human capital principals and teachers. The most robust partitions of the data are whether a principal was employed for both years of the experiment and fidelity of implementation of the management training.

In sum: they found impressive effects with talented principals who stayed in the job for two years but no effects overall due to principal turnover and too many low human capital leaders.

II. Why Did Fryer Need to Conduct an Experiment in Houston?

Data driven instruction and teacher feedback, which were key to the intervention, are not new ideas. Bambrick wrote Leveraged Leadership in 2012. And he surely wasn’t the first to implement these management practices.

So why did Fryer need to construct an experiment to apply these sound management practices in Houston?

Why wasn’t the Houston school district applying these techniques already?

As it happens, some other researchers (Nicholas Bloom, Erik Brynjolfsson, Lucia Foster, Ron Jarmin, Megha Patnaik, Itay Saporta Eksten, John Van Reenen) just published a paper on this very subject – with the aim of trying to understand the root causes of good management practices.

III. What are the Root Causes of Good Management Practices?

It’s hard to do a controlled experiment on management practices in the private sector, so caution is warranted in interpreting the results.

The authors used survey data and business results to determine whether sound management practices are correlated to increased business success (they are), and then tried to figure out what business conditions led to better management practices.

While the methodology is inherently tricky, it did reaffirm my priors.

The researchers found:

What could cause these huge differences in management practices across establishments? We found several major factors. First, establishments in more competitive industries (measured by the Lerner index) adopt more structured management practices. Second, those in more pro-business states (proxied by states with ‘right to work’ laws, as in Holmes 1998) tend to use more structured management practices. Third, establishments with more college graduates and firms located near universities (building on the work of Moretti 2004 for identification) tended to adopt more structured management practices. Fourth, being located near a successful large new entrant (using the ‘million dollar plants’ identification strategy of Greenstone et al. 2010) is correlated with more structured management practices, likely because it allows local companies to learn about practices from these large, successful firms.

All these factors matter, but they explained less than half of the variation in management techniques, which means that many other factors matter, too. One hypothesis is that individual managers and CEOs themselves are another critical driver (e.g. Bandiera et al. 2017).

To summarize: good management practices were most often found in (1) competitive industries (2) with less restrictive labor laws (3) located near universities and (4) successful new start-ups.

I know a city educational system that meets all these conditions.

It happened to achieve some of the best student achievement results the country has recently seen.

IV. Yes And

It if it ever occurs, it will take a few decades to scale the charter sector serve the vast majority of low-income students.

For this reason, I appreciate Roland’s efforts to see if charter practices can increase achievement in districts. While I don’t think this is the long-term game, there might be short-term benefits to be had.

But if you want these achievement gains to be sustained, you have to address root causes.

And the root cause of good management is not really about intellectually understanding good management practices.

Rather, it’s about creating the enabling conditions to sustainably execute these management practices.

I believe that non-profit governance will prove to be one of the most important enabling conditions in the public education sector.

More Money or More Charter Schools?

I review some of the recent research in a post at Education Next.

Here’s some math from the post:

Increasing Funding by Even 10% is Insanely Expensive

Consider a hypothetical town with 50,000 students, all of them who are in poverty, and a per-pupil allocation of $10,000.

Over ten years, increasing per-pupil by 10% will cost the town a half a billion dollars.

To put the costs in context: on average, it costs around $1,000,000 to launch a new charter school that serves 500 students.

This puts the cost of the charter intervention at roughly $100,000,000.

Also: the charter costs are one-time costs.

So over a ten-year period, the total bill for increasing funding by 10%: $500 million.

The total cost for scaling urban charters to serve all 50,000 students: $100 million.

For a fifth of the cost, you probably get 3-5X the achievement impact.

Do read the whole piece.