Category Archives: Research

Do all boats rise when charter schools expand?

I. A Hypothesis 

All organizations are founded on a hypothesis. Deliberate organizations are explicit about their hypothesis.

The City Fund’s hypothesis is that educational opportunity in cities will increase if:

(1) Non-profit schools enroll more students.

(2) Cities adopt a unified enrollment system to increase equitable access to all public schools.

(3) Elected officials encourage the best schools to expand and selectively transform struggling schools with new non-profit operators.

We don’t yet know if this is true, though early signs are promising.

Cities such as New Orleans, Denver, Newark, and Washington D.C. have seen strong gains using these strategies (as well as a focus on instruction and talent in district schools).

II. Fordham’s New Study on Charter Enrollment 

Fordham just put out a study that attempts to measure whether increases in charter enrollment in a city leads to all students learning more, including children in district schools.

Fordham found that, in urban areas, higher charter schools enrollment is associated with achievement gains for all black and Hispanic students in the city.

If it holds, this is an important finding on the benefit of expanding non-profit schools.

So how much weight should we give to the study?

On the positive side, the authors methodology is reasonable: they track a bunch of cities that are home to increasing charter enrollment, and then use a set of controls to try and determine if this increase in enrollment is associated with positive citywide results for minority students.

There are some clear limitations to this approach, most of which the authors acknowledge. The trickiest issue is causation: it’s hard to know if charter enrollment itself is causing the gains. For example, perhaps cities that see increasing charter enrollment also tend to be home to strong economic growth, and it’s the city’s economic gains that are driving better student performance.

Another major limitation is how much we can extrapolate from the cities in the data set.

Given that very few cities rapidly grew charters (ie, went from 10% enrollment to 50% enrollment), it’s hard to know how much we can draw from the study.

Perhaps citywide gains spike when charters increase from 10% to 30% (due to increased competition) but then reverse when charters go from 30% to 60% (due to financial pressures on the district). We won’t know until more cities reach higher charter enrollment.

III. What Can We Learn From the Study? 

The Fordham study should nudge us a bit toward the idea that increasing charter enrollment can increase learning for all students.

But, perhaps more importantly, it should cast serious doubt on the claim that the current rate of increased charter enrollment is significantly harming traditional public schools.

We can’t know if increased charter enrollment is causing citywide gains, but we can clearly observe that current charter enrollment is not causing major drops in district performance.

This is a very important finding. It refutes the major argument made by charter detractors.

This result mirrors some of what we’ve seen in CREDO’s recent analysis of city performance.

CREDO found an all boats rising effect in three of the most mature choice cities in the country. In Denver, Camden, Washington D.C., district schools improved as the charter enrollment increased.

It’s notable, though not dispositive, to us that these cities all have unified enrollment systems and transparent school performance information.

IV. How Can We Learn More?

Doug Harris and his team at Tulane are going to attempt a similar study but use a quasi-experimental approach. This should shed some more light on the issue.

We will also keep working with CREDO to hold the mirror up on the cities The City Fund is working most deeply with.

Lastly, it’s worth mentioning that understanding citywide impacts is important but not the only way to understand charter growth.

We should care a lot about the fact that charter school enrollment is increasing in the first place: it’s a clear sign that families are hungry for a better public education for their children. And that they view charter schools as way to meet their children’s needs.

Large scale correlational studies are not a substitute for simply observing that millions of parents are choosing charter schools in hopes of finding a great school for their children.

Why did New Orleans public schools improve so much?

Tulane researchers have a new paper that attempts to determine the causal mechanism for New Orleans school improvements.

A similar paper was written by Harvard researchers on the Newark reforms.

Both papers tried to answer the question: did things get better because schools opened and closed, or because existing schools improved?

Both papers come to the same conclusion: opening and closing schools is driving the gains in student learning (as measured by test scores).

The Tulane report came to a particularly strong conclusion. The authors write:

“The average school improved from the first to the second year after it opened, but school performance remained mostly flat afterwards… aside from the improvement when schools first opened, essentially all of the improvement in New Orleans’ average test scores has been due to the state regularly closing or taking over low-performing schools and opening new higher performing charters.”

The below graphic captures this finding in visual form:

Screen Shot 2019-08-29 at 12.45.18 PM.png

The authors end their study with a strategic recommendation and warning:

“The fact that newly opened schools continue to be better than those closed and taken over also suggests that the extreme measure of replacing school operators also still has some potential to generate further gains. At some point, the benefits from this strategy are likely to run out, but it does not appear that we have reached that limit yet.”

New Orleans has had strong government regulation over the past decade. For the most part, the best schools expanded and the government closed or transformed the worst schools.

It is an open question whether this good regulation can persist in New Orleans, or if it can be consistently scaled to other cities.

Of course, strong regulation is not the only way to shift enrollment to higher-performing schools.

A city could also simply let family choose amongst all schools and wait for lower-performing schools to fold under enrollment pressures. This process will be parent driven and likely slower.

Every city will need to figure out its own path when it comes to balancing top down accountability and bottoms up family choice.

Personally, I favor a combination of both. Let government have the ability to selectively transform the lowest performing schools in a city, and let families choose from a wide array of schools.

All boats rising in Denver public schools

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter, innovation, and traditional schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

CREDO’s analysis measures how much a school helps a student grow over the course of a year. They do this by comparing students in the city to similar students across the state.

The results just came in for Denver.

Denver Reform History

Over the past fifteen years, the locally elected school board partnered with superintendents Michael Bennet (now Democratic Senator for CO) and Tom Boasberg.

These leaders gave educators more freedom to tailor their school programs to the students they served. And they gave families more access to a diverse array of public schools. The district also made heavy investments in teacher and leader talent

This effort greatly expanded the number of public schools operated by non-profit organizations. Non-profit schools now serve around 30% of students in the city. These non-profit organizations are a mix of charter schools and innovation schools (district schools that operate with more freedoms under a non-profit board).

Superintendent Cordova just took the reigns last year.

CREDO Results: Every Sector in Denver is Outperforming Similar Schools Across the State

In Denver, traditional schools, charter schools, and innovation schools are all outperforming similar schools across the state.

The study’s author noted: “The pattern of performance here is consistent… it’s an incredibly strong advantage for students in Denver no matter what school they go to.”

A common critique of charter schools is they hurt traditional school performance. This critique has no grounding in evidence. And it does not seem to be true in Denver. All sectors in Denver are helping students grow.

Together, the sectors combine to achieve annual +.1 standard deviation effects in reading and math. These are large annual citywide effects.

Screen Shot 2019-08-20 at 11.06.12 AM

Where will Denver Head From Here? 

The Denver reforms have led to more than increases in test scores. High school graduation and college enrollment rates are also up.

Hopefully this will translate to more Denver students benefiting from Denver’s booming economy.

But for continued gains to occur, Denver should not abandon its most successful strategies. For so many kids in Denver, they have been a lifeline for increased academic learning.

More specifically: given that large academic gaps remain across racial lines, Denver would do well to expand those schools that are doing the most for kids of color.

Looking at 2018 data, these schools were rated highest in the city for closing academic gaps:

  • Polaris Elementary School
  • Slavens K-8 Schools
  • KIPP Northeast Denver Leadership Academy
  • Cory Elementary School
  • Steck Elementary Schools
  • DSST: Byers MS
  • Denver Green School
  • Stephen Knight Center for Early Education
  • Creativity Challenge Community
  • Holm Elementary Schools
  • DSST: Green Valley Ranch
  • Escalante-Biggs Academy

Hopefully these schools can grow to serve more students and help close persistent gaps across this city.

Good news for Camden’s children

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter, innovation, and traditional schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

CREDO’s analysis measures how much a school helps a student grow over the course of a year. They do this by comparing students in the city to similar students across the state.

CREDO presents its findings in standard deviations. A useful way to understand these impacts is to translate them into extra days of learning, based on a 180 day school year.

Screen Shot 2019-07-08 at 7.45.53 PM.png

As the chart above shows, a .15 standard deviation impact equates to about an extra half year of learning.

The results just came in Camden.

Camden Reform History

The state intervened in Camden schools in 2013. You can read more about the effort in this good New York Times piece.

One of the major innovations in the takeover was the creation and expansion of Renaissance schools. Renaissance schools are governed by non-profit organizations but must serve all students in the neighborhood. They are sort of a hybrid between charter schools and traditional schools.

The Renaissance reform effort was also coupled with improvements to traditional schools. Schools became safer and academic improvements were implemented across the city.

The city also created an online unified enrollment system to help families find the best public schools for their children.

Large Citywide Improvements

Camden’s city level effects are large.

In just two years, scores are up ~.15 standard deviations in math and ~.05 standard deviations in reading (compared to similar schools across the state).

To put this in context, over five years, New Orleans achieved a .4 standard deviation effect. These city effects were the largest the researchers had seen. Camden may achieve similar results. The math results are on track to mirror the gains seen in New Orleans.

It’s pretty incredible to see students learning so much more so quickly. Effects this large are a good signal that students are getting smarter in literacy and numeracy.

Screen Shot 2019-07-08 at 7.48.31 PM.png

 

Renaissance and Traditional Schools are Improving the Fastest

Renaissance schools are the highest performing sector in Camden, outperforming similar schools across the state in both reading and math. They also improved by over +.1 standard deviations in both subjects over the last year of the study.

The Camden traditional sector, though lower-performing, has improved. District schools have seen large improvements in Math (+.2 standard deviations) and modest gains in reading (+.06 standard deviations).

The charter sector continues to outperform the district, though it has seen a decline in its learning gains relative to the state over the past few years

Screen Shot 2019-07-08 at 7.50.41 PM

Screen Shot 2019-07-08 at 7.50.51 PM

Will Learning Improvements Lead to Better Life Outcomes for Children?

In New Orleans, we began to worry that gains in test scores, while important, would not translate into better life outcomes for students. Unfortunately, There were not enough post-secondary programs in the city that could help high school graduates prepare for meaningful careers.

Many cities across the country also struggle with this issue.

Recently, the former superintendents of New Orleans and Camden announced they were launching a new organization, Propel, to help high school graduates transition to good careers.

This promising effort, if it works, will help students capitalize on their increased numeracy and literacy skills.

Mission Not Accomplished

The Camden reforms are barely past their fifth year. The city is still home to struggling schools. Absolute achievement remains low. And the district remains under state takeover.

Hopefully, over the next five years the city’s schools will return to local control and continue to improve. And all of the work will translate to better life outcomes for students.

All public schools (traditional and charter) rising in Newark

When more students enroll in non-profit charter schools, what happens to the students who remain enrolled in traditional schools?

This is one of the most contentious questions in public education right now.

Past research has shown that increased public charter school growth does not negatively affect the academic performance of traditional public schools.

But much of this research covers geographies that don’t have that many charter schools.

An open question is whether the effects of charter growth on traditional public schools will change as charters serve more and more kids in a district.

In Newark, nearly 40% of students attend charter schools.

At this scale, non-profit schools have given families a lot more choices to find a good fit for their children.

But they have also put real academic and financial pressure on the traditional system.

So what’s been the effect?

New Jersey ranks all of its school districts based on academic performance. The state runs a couple of different types of analysis: ranking all district statewide and then also ranking districts based on those that have similar levels of poverty.

See below for the results for Newark citywide, Newark traditional system, and the Newark charter sector.

Newark’s Overall City Rank is Rising 

Newark has shot up from the 39th percentile to the 78th percentile amongst the thirty-seven highest poverty districts in New Jersey,

In the 100 highest poverty districts, Newark has moved from the 18th percentile to the 50th percentile.

When comes to all districts, Newark performs poorly, though it has made major progress in the past five years. This progress comes after a fairly long period of stagnation.

Screen Shot 2019-06-17 at 9.12.00 AM

Newark Traditional Public Schools are Improving at a Healthy Rate

Newark’s traditional schools have made major improvements over the past five years, after being fairly flat in previous years.

When it comes to the highest poverty cities, the district’s traditional public schools have moved from the 20th percentile to at or above the 50th percentile.

They also have also seen gains the other performance rankings, though overall gains when compared to all New Jersey districts are fairly modest.

Screen Shot 2019-06-17 at 9.12.23 AM

Newark Charter Schools have Nearly Caught the State Average 

Newark charters are achieving at very very high levels.

Taken together, they are the top performing high-poverty district in the state.

Even more impressive, Newark’s charters have risen to nearly the 50th percentile in the entire state of New Jersey.

New Jersey is one of the wealthiest states in the nation.

Students in Newark charter schools, who are mostly children of color living in poverty,  are performing as well as their much more affluent and privileged peers.

Screen Shot 2019-06-17 at 9.12.10 AM

Still an Open National Question

Newark is just one of many cities that are now home to large non-profit public school sectors.

We are working with researchers to study city level and sector level effects across many of these cities.

In the fall, I hope to have a more comprehensive write-up on these results.

But, for now, it’s good to see all schools rising in Newark.

Hopefully these results will hold true across many more cities.

Bloom, New Orleans, and Effect Sizes

Six months ago Matthew Kraft published an excellent article on effect sizes.

I worked in education for five years before I had any understanding of research design and reporting. I wish Matt’s piece was around a decade ago.

His article is a bit dense if you’re trying to just wrap your head around the issue, so consider this post a lay person’s intro to Matt’s piece and the subject itself.

If you catch any mistakes, please do let me know. I’m still learning.

Why are effect sizes useful?

Consider currencies. Currencies are useful because they allow you to easily compare prices across various goods. Instead of having to constantly refer to one set of goods in relation to another set (ie, three apples are worth the same as four oranges which is worth the same as three paperclips), we can use the same unit (dollars) to compare a bunch of different goods.

Effect sizes serve the same function. They help us easily compare the magnitude of the impact of a bunch different interventions. We can do research on graduation rates, test scores, suspension rates, or whatever we want, and then we can convert our results into an effect size to help us compare how big of an impact we had.

Effect sizes are the unit of currency for measuring impact.

What is an effect size?

Many effect size calculations in education research are expressed in standard deviations.

A common formula to determine the effect size is:

(mean of experimental group – mean of control group) / standard deviation

Let’s say we trying to find the effect size of a new math curriculum on test scores. We might give half the population the new curriculum, half the old curriculum, and then see what the difference is.

Let’s say the difference is +5 pts out of a 100 for the students using the new curriculum. The curriculum “worked.”

But what does that mean?

We now want to know if +5 pts is a big deal. This is where the standard deviation comes in.

A low standard deviation means there is very little difference in the population (everyone is scoring about the same score). A large standard deviation means there is a wide spread in scores.

Because the standard deviation is the denominator in the formula, the smaller it is, the large the effect will be for any given difference between two groups.

In other words, if everyone is scoring between 62 and 65 out of a hundred, and you jump five points, you could go from the bottom 1% of test takers to the top 1% of test takers.

Because the standard deviation is low (small spread), a modest jump leads to a big effect.

What is a large effect?

This is where Matt’s paper is particularly useful.

Much of the previous literature on effect sizes made many mistakes:

  1. Sample sizes were ignored.
  2. Duration of treatment were ignored.
  3. Time elapsed until measurement was ignored.
  4. Cost was ignored.

Taken together, scalability of interventions was ignored. This had the unintended consequence of setting the bar too high for what should be considered a large effect size.

Bloom’s 2 standard deviation effect 

You may have heard of Bloom’s 2 sigma tutoring intervention. This result is taken to show that 1-1 tutoring can have a 2 standard deviation (very large!) effect.

But Bloom’s study design was the following: take dozens of 4th, 5th, and 8th graders; give them 1-1 tutoring in discrete subjects like cartography or probability; and then test them on what they learned after 3-4 weeks!

It’s much easier to squeeze out a big effect under these conditions.

These types of small sample studies led to a research norm where an effect size had to be .8 standard deviations for it to be considered large.

New Orleans’ .4 standard deviation effect 

Contrast Bloom’s study to Doug Harris’ study on the New Orleans education reforms.

The New Orleans study covered tens of thousands of students. Students received the treatment across all major subjects, including math, reading, science, and social studies. The treatment lasted multiple years. And students were tested once every year in each subject.

It’s a lot harder to make large gains under these conditions, especially when the intervention costs under 20% of 1-1 tutoring.

Doug’s study found .4 standard deviation effects for New Orleans students over a five year period.

In his paper he wrote that he was “not aware of any other districts that have made such large improvements in such a short time.”

To summarize:

  1. The standard bar for a large effect was .8 standard deviations. This was irregardless of sample size, length of treatment, measurement proximity, or cost. The bar was poorly constructed.
  2. New Orleans achieved a +.4 standard deviation effect on test scores.
  3. Researchers had never seen a citywide effect this large before.

There are two ways to interpret this.

  1. The previous .8 standard deviation bar was way too high for large samples.
  2. The New Orleans effect, despite being relatively large for district improvement, is still so absolutely small that we should not be too impressed.

Was the New Orleans effect too small?

The +.4 standard deviation effect equates to the average New Orleans student moving from the 22nd to 37th percentile in performance.

For any individual, this might or not be life changing. But in the aggregate this means the average New Orleans student roughly went from a borderline high school dropout (bottom 20% of performance) to a student who has a real chance to enter a two year or four year college (modestly below average performance).

Across a large population, this is a pretty big deal.

We should pay attention to a city level +.4 standard deviation increase in test score. If this effect (or even one somewhat lower) can be scaled, kids across the country will have a better chance at leading a good life.

Of course, academics and test scores are just one piece of the puzzle of economic mobility, but they are an important piece. Schools with negative effects on test scores tend not to deliver great long-term life outcomes for kids.

Matt Kraft’s proposed effect size scale

When it comes to large interventions, Matt argues we should get rid of the .8 standard deviation benchmark.

I agree.

Matt proposes the following rough scale:

Small effect: less than .05

Medium effect: .05 to .2

Large effect: .2 or larger

Matt reviews a bunch of educational studies to help come up with this table. While I don’t love that it averages a bunch of very different studies, at the very least it sets conservative estimates on effects and cost (given that averages include studies that don’t meet the highest bar for sample / duration / etc.).

Screen Shot 2019-05-29 at 5.10.56 PM.png

Take a look at where .4 standard deviations shows up. The New Orleans reforms are in the 90th percentile of magnitude but the 60th percentile of costs. New Orleans increased it’s pup-pupil by $1,400 in the years following Katrina, though it’s not clear to me that the money is what really drove the effect. But even if you assume it did, the results pass a ROI test.

Again, the New Orleans impacts are pretty remarkable.

In considering impact, cost, and scale, Matt also provides the following matrix:

Screen Shot 2019-05-29 at 9.49.25 PM.png

New Orleans does well.

In Sum: Toyotas > Ferraris 

When it comes to effect sizes, be very careful to review sample sizes, treatment duration, measurement proximity, and cost.

Holding out for .8 standard deviation effects is foolish. These effects will rarely occur and when they do they tend to be very hard to scale.

When it comes to large scale interventions across medium term time frames, effects above .2 standard deviation warrant our attention.

The most realistic path for broad academic gains is to look for meaningful jumps in student performance that are caused by an intervention that has a real chance of scaling over time. And then testing and scaling and testing and scaling.

In other words: Toyotas > Ferraris.

Beware of under powered studies

Mathematica just released a study on how charter middle schools impact college enrollment and completion. Before I dig into the study, some quick context.

The research base on the connection between test scores and life outcomes

An important research finding over the past decade has been that schools with neutral test score impacts sometimes achieve positive lifetime outcomes for their students.

Equally important, schools with positive test score impacts often don’t achieve positive life outcomes for their students.

Another important finding (a finding that is often glossed over by my libertarian leaning friends) is that schools with negative test score impacts are much less likely to achieve positive lifetime outcomes for their students.

Taken together, these findings have narrowed my belief in test based accountability. Previously, I put a lot of weight on the difference between positive, neutral, and negative test score schools.

Now I focus more attention on selectively transforming schools with negative test score impacts.

I also care much more about parent demand than I used to.

The Mathematica Study Limitations 

The Mathematica study has two very major limitations.

First, it covers charter schools from a previous era (2005-2007). In the subsequent decade, many charter schools (especially those we work closely with) have increased their focus on getting kids to and through college and into meaningful careers.

Second, the study is very small. The sample only included 10 urban charter schools. Additionally, only 3 schools in the entire study had poverty rates over 75%!

The small sample means the study is fairly under powered. With a sample this small, only large effects will be picked up. A modest but positive correlation between achievement and college graduation, for example, would not be statistically significant in this study. 

Other similar studies have looked at entire states, and included many more high poverty charter schools, so I place much less weight on this study.

Findings

The study did not find a statistically significant relationship between test score and college completion results. The chart below details the major findings, with each dot representing a school.

College Completion 

mathematica 2

When you eyeball the graph, some trends emerge, especially the low number of schools in the bottom right quadrant (high test effects, low college effects). 

When I emailed the authors about this trend, they said they were not statistically significant and we should be cautious to draw conclusions by looking at patterns the scatter plot. 

In Sum

The small sample size and the decade old study window period mean that the study itself won’t move my opinion on the issue that much.

Only 3 schools in the study were high poverty schools. Nearly all of the schools that we work with are high poverty schools.

In partnership with the Arnold Foundation, we’re supporting a bunch of more research on the issue, and I’m excited to learn more. 

I will change my mind if the findings change.  

 

 

Julia Galef interviews me on the Rationally Speaking podcast

I just did a podcast with Julia Galef of Rationally Speaking.

You can listen here.

We cover: New Orleans, the evidence base behind urban charter schools, common critiques of charter schools, how to balance evidence and lived experience, what we do know and don’t know about what happens when charters expand, what we do know and don’t about what parents want…. and more.

Here is a teaser:

Juli: Right. Has anyone yet proposed a nudge system, where we try nudge parents to pick the schools that actually improve scores more?

Neerav: There are experiments being run right now about how you show information to parents in these systems, and figuring out what that means. Now, again, you want a balance. Do you know what the right thing to show is? Because if you’re nudging, that means you have an opinion on how parents should be making decisions.

Julia: Right.

Neerav: Versus, do you want to be a little more humble and say, “We’re just gonna show the information as neutrally as we can and let you know”? Those are pretty complicated decisions for policymakers, on how to design choice systems.

Julia is an excellent interviewer.

If you have 30 min at the gym, on a walk, or during your commute – give it a listen.

Being less racist and sexist may account for 25% of increased economic output in the United States over past 50 years

Two of the most important goals of a liberal society are to treat people fairly and to increase economic productivity.

Treating people fairly increases the chance that individuals and communities can  flourish.

Increasing economic productivity usually leads to better overall health, wealth, and happiness.

It’s possible for these two goals to be at odds.

When it comes to racism and sexism in America, the goals don’t seem at odds.

This recent paper found that the United States saw major economic gains by being less racist towards African-Americans and less sexist toward woman.

It used to be the case that most leading professionals were white men. This is less true today.

The authors of the paper explore what we can draw from this change:

 In 1960, 94 percent of doctors and lawyers were white men. By 2010, the fraction was just 62 percent. Similar changes in other highly-skilled occupations have occurred throughout the U.S. economy during the last fifty years. Given the innate talent for these professions has unlikely changed differentially over time across groups, the change in the occupational distribution since 1960 suggests that a substantial pool of innately talented blacks and women in 1960 were not pursuing their comparative advantage. We examine the effect on aggregate productivity of the remarkable convergence in the occupational distribution between 1960 and 2010 through the prism of a Roy model. About one-quarter of growth in aggregate output per person over this period can be explained by the improved allocation of talent.

The paper details some powerful anecdotes to make their case.

For example, Supreme Court Justice Sandra Day O’Connor graduated from Stanford Law School in 1952; despite being ranked third in her class, the only private sector job she could get after graduating was as a legal secretary.

In teasing out what specific changes made things better, the authors consider three possibilities:

Changes in preferences; ie, there wasn’t much discrimination; it’s just that African-Americans and woman increasingly wanted more professional careers post 1960.

Reduction in discrimination in preparing and entering a field: ie, there were barriers that prevented talented people from going to school and entering specific fields.

Reduction in discrimination once in the field; ie, there were a bunch of glass ceilings that prevented advancement.

The authors find that declining barriers to entry into a field explain 24 percent of growth in U.S. GDP per person between 1960 and 2010.

Declining labor market discrimination once in the field explains 6 percent of growth.

Changing  preferences across groups explain little of U.S. growth during this time period.

So it wasn’t that after 1960 a bunch of African-American and women suddenly wanted to become doctors.

Rather, things were bad because a lot of white men prevented African-American and women from becoming doctors.

All the usual caveats apply: it’s one paper on a topic that can probably never be fully understood through pure academic research.

But it’s a useful reminder that being less racist and sexist has both individual rights and economic benefits.

I think increasing educational opportunity, in this sense, is akin to reducing discrimination.

If educational opportunity is further increased in our country, we’re likely to see major gains on both moral and economic fronts.

Rearranging how parents get information about schools may increase educational opportunity at no cost

A .2 standard deviation increase in academic performance is a pretty good sized effect.

In this study, the .2 effect equates to attending a school that is 5 percentile points higher in ranking in academic performance.

Most interventions that achieve a .2 effect cost money.

This intervention costs nothing.

Unified Enrollment Systems 

Over the past decade many cities have adopted unified enrollment systems. These systems allow families to go online and view information on all public schools in their city, and then submit their ranked preference of schools to the government. New Orleans, Indianapolis, Denver, Chicago, Newark, Camden, and New York all have some version of this system for at least some grades.

These systems are great in that they give parents more information, allow them to easily apply to schools online, and help policy makers get information on which schools are most in demand by parents.

I’ve previously written about how the user interfaces for these systems diverge greatly in in quality. Some feel like you’re using a great iphone app and some are barely better than opening PDF files.

How Does User Interface Affect School Selection?

In this study, researchers worked with a consumer testing company to recruit a group of parents to use a generic unified enrollment system to select a school.

They then broke the sample into groups and presented a different user interface to each group, with the aim of testing how presenting information would impact school selection.

Here’s what they found:

Screen Shot 2018-11-07 at 8.27.24 AM

You can see the biggest effect (.19 standard deviations) is in the row “default sort order” and the column “distance.”

The researchers created two default sort orders: in the first case, you put in your address and then you are shown the schools nearest to your address; in the second case, you are shown the highest performing schools available to you.

The researchers found that if you make academic performance the default sort order, parents ended up picking schools that were +5 percentile points higher ranked on academic performance.

Making academic performance the default order costs no money.

Limitations 

The study has some real limitations.

First, the stakes weren’t real. The parents weren’t actually selecting a school that they would send their child to. This probably meant they put less effort into the school selection. They also weren’t able to get information from other sources (like friends and family).

Second, having parents pick schools with higher academic performance ratings is only useful if those ratings accurately measure student learning. In cities that use value-added methods for school rankings, I’d feel more comfortable with this nudge. In cities that mostly use absolute test scores, I’d feel less comfortable.

Third, academic performance isn’t everything, and parents select schools for a variety of reasons. Ultimately, designers of the interface do have to make choices, so I’m ok with a bit of nudge toward academic performance, but I don’t think we should make this nudge at the expense of giving families a wholistic picture of schools.

All that being said, the study shows that small and easy to make changes in user interface may have an impact on how families select schools.

If I was a government official that managed a unified enrollment system, this study would lead me to experiment with similar interventions for my own city’s system.

At the very least, I’d want to make sure that my user interface decisions were deliberate and values based rather than ad-hoc and random.

Access and Supply

Lastly, great unified enrollment systems are about equalizing access to great public schools.

They do nothing to increase the number of great public schools.

Cities would also do well to do all they can to help their best public schools expand.