A friend just emailed me some questions about the RAND personalized learning study.

After reading the study, I too had some questions. My initial read is that the report did not do an effective job in drawing the right conclusions from the data.

Perhaps I’m misreading the study. The folks at RAND generally produce solid research. If I made some mistakes in my analysis, I’ll update the blog to reflect my errors.

Overview of Study Design: 

• The treatment group implemented personalized learning strategies.
• The treatment group included charter schools and district schools that received funding through a competitive personalized learning grant process.
• The control group is made up of matched students and schools with similar performance,  demographics, and governance conditions.

Results:

• Charter schools, in aggregate, achieved very significant and positive results: .1 to .4 effects.
• District schools, in aggregate, achieved no positive results (the sample size was also smaller).

Concern #1: What is the Intervention?

The title of the study is “promising evidence on personalized learning.” But how do we know that personalized learning is driving the gains?

Those familiar with CREDO’s research on Charter School Growth Fund will remember that their portfolio achieves a ~.2 effect.

The charters in this study are hitting about the same marks, give or take.

So are we seeing promising evidence on personalized learning or further evidence on the effect of high-performing charter schools?

Running personalized learning charter schools through a rigorous selection model and then comparing them to other schools (even if they are schools of choice) is not a way to tease out the effect of personalized learning.

A better design would have had some high-performing charters adopt personalized learning and then compare their results to other high-performing charter schools.

Without this type of comparison, I don’t think we know that personalized learning is driving the gains; rather, the gains could just be caused by other elements of high-perfoming charter schools (culture, data usage, teacher coaching, etc.).

Concern #2: Is the Lede Buried?

The fact that district schools didn’t achieve positive results was the sixth “Key Finding” listed in the report, and it was not mentioned in the RAND website overview of key findings.

This happens too often.

Whether we’re talking about Newark, personalized learning, or some other topic, many times:

1. Charters deliver most of the gains.
2. The gains are reported out as positive.
3. The fact the district didn’t improve (or improved much less) is not mentioned or is buried.

Yes, there are some exceptions, such as Washington D.C., where the district is also delivering real gains.

But too often charter effects are reported out as general effects.

In Sum

It’s great to see charters implementing personalized learning and getting strong results.

It’s disappointing to see that these gains were not achieved in district schools.

For me, this is further evidence that governance might be the most effective intervention we can deliver to ensure that all kids get the schools they deserve.