A .2 standard deviation increase in academic performance is a pretty good sized effect.
In this study, the .2 effect equates to attending a school that is 5 percentile points higher in ranking in academic performance.
Most interventions that achieve a .2 effect cost money.
This intervention costs nothing.
Unified Enrollment Systems
Over the past decade many cities have adopted unified enrollment systems. These systems allow families to go online and view information on all public schools in their city, and then submit their ranked preference of schools to the government. New Orleans, Indianapolis, Denver, Chicago, Newark, Camden, and New York all have some version of this system for at least some grades.
These systems are great in that they give parents more information, allow them to easily apply to schools online, and help policy makers get information on which schools are most in demand by parents.
I’ve previously written about how the user interfaces for these systems diverge greatly in in quality. Some feel like you’re using a great iphone app and some are barely better than opening PDF files.
How Does User Interface Affect School Selection?
In this study, researchers worked with a consumer testing company to recruit a group of parents to use a generic unified enrollment system to select a school.
They then broke the sample into groups and presented a different user interface to each group, with the aim of testing how presenting information would impact school selection.
Here’s what they found:
You can see the biggest effect (.19 standard deviations) is in the row “default sort order” and the column “distance.”
The researchers created two default sort orders: in the first case, you put in your address and then you are shown the schools nearest to your address; in the second case, you are shown the highest performing schools available to you.
The researchers found that if you make academic performance the default sort order, parents ended up picking schools that were +5 percentile points higher ranked on academic performance.
Making academic performance the default order costs no money.
The study has some real limitations.
First, the stakes weren’t real. The parents weren’t actually selecting a school that they would send their child to. This probably meant they put less effort into the school selection. They also weren’t able to get information from other sources (like friends and family).
Second, having parents pick schools with higher academic performance ratings is only useful if those ratings accurately measure student learning. In cities that use value-added methods for school rankings, I’d feel more comfortable with this nudge. In cities that mostly use absolute test scores, I’d feel less comfortable.
Third, academic performance isn’t everything, and parents select schools for a variety of reasons. Ultimately, designers of the interface do have to make choices, so I’m ok with a bit of nudge toward academic performance, but I don’t think we should make this nudge at the expense of giving families a wholistic picture of schools.
All that being said, the study shows that small and easy to make changes in user interface may have an impact on how families select schools.
If I was a government official that managed a unified enrollment system, this study would lead me to experiment with similar interventions for my own city’s system.
At the very least, I’d want to make sure that my user interface decisions were deliberate and values based rather than ad-hoc and random.
Access and Supply
Lastly, great unified enrollment systems are about equalizing access to great public schools.
They do nothing to increase the number of great public schools.
Cities would also do well to do all they can to help their best public schools expand.