Category Archives: Predictions

Could Newark have achieved more?

Harvard researchers just published a study on Newark.

In large part due to Mark Zuckerberg’s $100 million gift to the reform efforts, the city has seen a lot of coverage, including the well done book The Prize.

Most of this coverage has been negative.

The Harvard Study: Teasing Out Annual Effects From 2010-2016

The Harvard study found that the Newark reforms, in the most recent year of the study, had a positive impact on ELA (.07 SD) and no impact on math. In the early years of the reforms, the achievement effects were negative.

Unfortunately, the researchers only looked at annual effects (were there positive or negative effects in any given year) rather than calculating cumulative impacts (what was the total impact over these years). Because of this, we can’t perfectly compare the Newark results with results of other city based studies, such as Doug Harris’ evaluations of the New Orleans efforts.

What Caused the Positive Effects? 

While the authors didn’t calculate the cumulative effects of the reforms, they did do something wonderful.

The authors separated out the effects of two different strategies: (1) improving existing schools vs. (2) expanding high-performing schools, closing low-performing schools, and facilitating the transfer of students out of low-performing schools and into high-performing schools.

The improving existing schools approach included replacing large numbers of principals, renegotiating the union contract, implementing new data systems, and extending learning time.

The open / close / shift enrollment approach included adding to the enrollment of high-performing charter and district schools, closing underperforming schools (11 traditional schools and 3 charters), and implementing a unified enrollment system that made choosing schools much easier for families. These reforms increased charter enrollment from 14% to 28%.

The results are striking.

In math, the improve strategy achieved a .08 negative effect in math while the open / close / shift strategy had a positive .04 effect.

In ELA both strategies had positive effects, but the open / close / shift was responsible for 62% of the overall positive effect.

Perhaps most importantly: the open / close / shift strategy achieved positive effects in every year of the study. 

Opening and closing schools, and shifting student enrollment, increased student learning.

Could Newark have Achieved New Orleans’ Level Gains?

In roughly the same number of years as the Newark reforms, New Orleans achieved positive .2-.4 standard deviation effects.

New Orleans, of course, relied much more on the open / close / shift than it did on improvement.

As noted above, we can’t do a direct apples-to-apples comparison of the studies. But I’d be surprised if Newark’s cumulative effect is as large as the New Orleans effect given that Newark saw negative effects in 3 of the 5 years of reforms.

We’ll never know if Newark could have achieved similar gains to New Orleans, but in many ways the Newark circumstances were more enviable than those of New Orleans.

Newark is home to one of the highest-performing charter sectors in the country, while New Orleans had no large effective charter operators in the city right after Katrina.

Newark’s initial philanthropic investment was much larger than that received in New Orleans after Katrina. And the public per-pupil in Newark is about 3x that of New Orleans.

Newark students also didn’t have to go through the immense trauma of Katrina.

That being said: perhaps New Orleans style reforms would have not worked in Newark.

Maybe the politics would not have allowed for it… though it’s worth remembering that Shavar Jeffries, now the CEO of Democrats for Education Reform, was only about 2,000 votes away from beating Ras Baraka in the 2014 Newark mayoral race.

Or perhaps the charter sector would not have been able to grow much faster than it did; notably, charter effects did decline as they enrolled more students… though perhaps more charter capacity would have emerged under better conditions.

All told, my guess is that Newark did not increase educational opportunity as much as it could have.

A 2010 Prediction 

On this blog, I try to be honest about both my mistakes and successes.

It would be disingenuous for me to write a post arguing that, in 2010, no one could have predicted which strategy would have been better.

The fact is that I did predict this. In 2010, when it became clear that there was going to be major reforms in Newark, I created a powerpoint deck to give advice to national educational leaders. In the deck, I modeled out how quickly I thought you could expand great schools and close failing schools.

Given the relatively small size of Newark public schools (50,000 students), as well as the number of high-quality charter schools, I felt confident that the vast majority of Newark students could attend a high-quality school within ten years.

This ended up being the strategy we took in New Orleans.

It was not the strategy taken in Newark.

Learning 

I have a lot of love for the philanthropists, political leaders, and educators who spent the good part of a decade trying to make things better for kids. I know many of them well and have had nothing but positive interactions.

In terms of rigorous third party research results, we also knew a lot less in 2010 than we do now. There have been a wave of recent studies documenting the positive effects of expanding the best schools and closing the worst.

And while we still suffer from a dearth of research on citywide turnaround efforts, outside of the early results in Lawrence, Massachusetts, we really haven’t seen a citywide turnaround reform effort significantly raise achievement without a focus on opening and closing schools (research has shown that Washington D.C. improved their public schools, but the city also leaned heavily on opening and closing schools).

So perhaps the better question is not what should have Newark leaders done in 2010, but what should similarly situated civic leaders do in 2017?

Civic leaders should embrace strategies that maximize the expansion of the best schools and close the worst schools.

Of course this need not be the only focus of a city turnaround effort, but the Newark study has increased my belief that it should be the primary focus.

What if unified enrollment platforms were 10x better?

An emerging group of cities – including Washington D.C., Newark, Camden, New Orleans, and Denver – have adopted unified enrollment systems. With these systems, families can enroll in schools across the city via an online application system.

This is a huge step forward. For too long, parents have not had enough information or access to the public schools in their cities.

However, the new enrollment systems are still in their infancy. The best version of these systems could radically improve public education. Unfortunately, we’re very far from this endgame.

I. Early Wins: Access, Equity, and Ranking

Access: With the best open enrollment systems, families who can’t afford a house in a fancy neighborhood can now finally transparently apply to a school in a more wealthy neighborhood.

As a result of increase in access, a recent study in Washington D.C. found that the new enrollment regime would likely reduce segregation over time:

Screen Shot 2017-03-17 at 9.23.22 AM

Equity: In most cities, traditional and charter schools play a ton of games with enrollment. Traditional magnet schools use opaque entry requirements. Bad apple charters don’t take in kids with special needs. There is no equity.

With online enrollment platforms, these problems go away, as schools are no longer in control of their enrollment.

Quite simply: the algorithm is fairer than the enrollment clerk.

Ranking: These new enrollment systems also allow parents to rank their top schools. This is extremely important.

First, a family’s high desire to enroll their child in a school can now  be translated into an increased chance that they actually get into this school.

Previously, high desire meant little unless you were connected, wealthy, or dogged.

Second, ranking allows  parents to publicly signal to government which schools are most and least in demand (which will ideally affect opening, expansion, and closure decisions). It also signals to school operators what attributes make a school in high demand.

By analyzing ranking preferences, researchers in New Orleans were able to correlate school characteristics with parent preference:

Screen Shot 2017-03-17 at 9.44.39 AMRanking transforms family desire into actionable information.

II. Unified Enrollment Systems are Mediocre Platforms

In preparation for writing this blog, I spend an hour on unified enrollment system websites. It was not a great experience.

Here is the school finder homepage from Washington D.C. – I couldn’t even find a way to filter schools by academic performance!

Screen Shot 2017-03-17 at 9.50.14 AM

Here is the search function for Newark’s enrollment system – you have to download a pdf!

Screen Shot 2017-03-17 at 9.55.36 AM

By comparison, here’s the search page from Zillow:

Screen Shot 2017-03-17 at 9.53.10 AM.png

On Zillow, you can easily search for homes based on the most useful search criteria. Yes, Zillow is surely better capitalized than your average enrollment system, but even with modest funds a city should be able to do better than a downloadable pdf.

III. Moving From Equity and Ranking to Matching and Prediction

More sophisticated uniform enrollment could offer two extraordinary improvements: they could better match families with schools, and they could better predict how any given student would do at a school.

Matching: Right now families mostly use enrollment systems for ranking: they know the schools they want and they use enrollment systems to express this desire.

What is not really happening (as far as I can tell) is sophisticated algorithms actually helping families match with schools.

For example, on Camden’s enrollment site (where you can thankfully filter by academic performance!), I found three schools that all met the “on track” performance criteria, and pulled up the comparison page:

Screen Shot 2017-03-17 at 10.11.14 AM

This does very little to help me choose between these schools. My preference variables are limited to very broad categories such as “art classes” and “after school programs.”

After playing around on these websites, I get very little indication that that the platform knows me or the schools very well. Compare this to Netflix, Amazon, or dating websites (previous life) – platforms where I feel like the algorithms know me better than myself.

Unified enrollment systems need to more deeply understand children and schools in order to make better matches.

Prediction: Right now, government accountability systems are a basket case of poor design (generally don’t weight growth enough), brutal politics (what politician wants to tell a bunch of communities they only have “F” schools?), and awful transparency (good luck trying to navigate your average state department of education website).

Most importantly, government accountability systems evaluate schools rather than make predictions.

As a parent, it’s one thing to tell you that a school is a “C+” – it’s another thing to give you a prediction of what will happen to your child if she attends the school.

With current date, we could probably gather basic information on your child’s age, gender, current academic performance, personality type, etc.,  and make a reasonably accurate prediction that if she attends school X she will have a Y% chance of graduating from high schools and a Z% chance of earning a post-secondary degree.

Good enrollment systems, over time, should become better and better predictive agents, and, perhaps, can end up augmenting (displacing?) government accountability systems.

IV. Root Causes and Potential Solutions 

I don’t yet have strong beliefs about the root causes of why these enrollment products aren’t getting better faster. But here’s some guesses:

Non-profits > government operated: Most of the enrollment systems are run by governments, which are not good at running tech products and have bad incentives around giving parents accurate information about schools. Non-profits would likely be better operationally and have better incentives, and avoid the privacy concerns associated with for-profits.

Lack of scale: Matching and predication can better with bigger data sets, and if all these systems are structured as isolated city based data silos, the algorithms will be dumber than they should be.

Weak Customer Demand -> Bad Economics: SchoolMint, from what I understand, is the most successful player in the market. For reasons I don’t underhand, this company has not developed a better product. Perhaps it’s because their government customers don’t actually want it. Or perhaps the economics don’t work (which might suggest philanthropy is needed).

If the above is true, a national non-profit should be backed to scale to enough size to create smart algorithms, and it should be financially structured in a manner that gets it out of the perverse incentives of being beholden to government or individual schools rather than families.

A philanthropic foundation with a great tech backbone could be well situated to support this endeavor.

V. Expectations

Better matching and prediction would probably not make the average student’s education experience 10x better, just as dating websites don’t inevitably lead to great marriages.

But I do think better matching and predication could increase the probabilities that millions of families could find a better fit for their children.

At scale, that’s a better world.

6 Numbers to Watch in 2015

prediction ball

In the blogging world, it’s become obligatory to put out yearly predictions.

While I have thoughts on longer-term trends, I have no idea what will happen in any given year.

That being said, here’s numbers I’ll be watching to better understand where the long-term trends are heading.

1. Math Software Program Effects Sizes

We’re starting to see math software achieve .1-.2 effects. If these effects continue to hold-up, and they do so on rigorous math content, I’ll grow more bullish on tech based math instruction.

2. Average SAT Score of New Teachers

Research is showing that new teachers are increasingly scoring better on aptitude tests, which probably bodes well for long-term teacher effectiveness. I’m curious whether this will hold as the effects of the recession recede.

3. National Charter School Market Share

Over the past couple years, charter market share has grown between 7-10% annually. Given that millions of children attend charter schools, this pace of growth is very significant. I’m interested to see if the growth rate holds as the absolute number of students continues to increase.

4. Number of Urban Charter Markets with +40% Penetration

Urban markets should be viewed as distinct from overall national charter market share, as this is where the best charter work is occurring. In the decade, my hunch is we’ll have over ten urban markets that are majority charter. This evolution will began to call into question the very structure of public education in major cities. So I’m curious to see what progress we make on this mark in 2015 (2014 market share data here).

5. Number of States with Rigorous Assessments 

Note that I didn’t say standards, which are useless. Nor did I say PARC or Smart Balance, which are simply two types of rigorous assessments. The number I care most about is how many states implement some form of rigorous assessments. Given that I do believe assessments impact instruction, I’m eager to understand how much the Common Core push will lead to more rigorous assessments, in some form or another.

6. Teacher Union Membership

Teacher unions have been losing members. Relatedly, the number of teachers covered by collective bargaining agreements continues to drop. If this trend continues for the next decade, will likely impact the politics of education reform.

In Sum

Long-term trends in tech effectiveness, teacher quality, education governance, state assessments, and union membership will all affect the future of education in this country.

It’s worth keeping an eye on these numbers.