Category Archives: Regulation

Most students in NOLA whose school is closed end up in one of their top choices for the following year. Here’s how.

This post from Ed Navigator is worth reading. It covers schools closures in New Orleans.

Over ten years after Katrina, and under an elected school board, New Orleans continues to selectively close underperforming schools.

I view this as a good thing, given the growing body of research that shows that school closures help kids when the students end up in better schools.

New Orleans uses a unified enrollment system to help kids get into better schools.

The unified enrollment system gives preference to students whose schools were closed the year before. If your school was closed, the algorithm bumps you to the top of the list for any school you want to get into.

Ed Navigator works with families whose schools have been closed, so that they can help select great schools.

The result?

This year, 87% of students who attended a closing school and used the enrollment system received on of their top three choices for the next school year.

94% of the students will now attend a school that is rated higher by the state’s grading system.

The system is by no means perfect. My biggest critique is that the state’s grading system still relies too heavily on absolute test scores (rather than growth). I also understand the counterarguments that government should never close schools and should instead let enrollment patterns (driven by parental choice) determine which schools grow and which close.

But I would rather have the New Orleans enrollment and closure system than just about any other big city system in the country.  In too many cities, really bad schools stay open for too long. And if anything happens to them, kids often end up in schools that are just as bad.

This is not what happens in New Orleans.

It’s also great to see Parag Pathak (and his colleagues) work in action. Parag recently won the John Bates Clark award in part because of his contributions to working on unified enrollment systems.

It’s rare that an idea goes from the ivory tower to think tanks to actual implementation by a democratically elected body to  helping citizens.

This is really great to see. And really great for kids in New Orleans.

Response to Matt Ladner and Jay Greene

Over at Jay’s blog, Jay and Matt wrote two critical posts of portfolio management and harbormasters (our team calls them quarterbacks).

Jay and Matt think differently than I do, have different political orientations, and are sharp. Their writing makes me smarter.

I also find their post titles, writing tone, and evidence analysis to be a bit over the top. They sometimes overstate their claims and under appreciate the other side of the argument.

In these two posts, Jay and Matt use NAEP charter sector gains in Arizona, Michigan, and Texas – as well as the mediocre NAEP scores seen in Louisiana’s charter sector – to argue that portfolio management and quarterbacks aren’t working.

I found their analysis to be overly narrow. Instead of taking some new evidence in and synthesizing this with the broad set of evidence available, they anchored on to one set of data points and made too strong of claims (especially in the titles of their posts).

Don’t Look at NAEP in Isolation 

Matt is right to point out that some states with fairly loose charter regulations saw a lot of charter gains in NAEP between 2009 and 2017.

I think this should modestly increase our belief that being loose on charter openings and closings can lead, over time, to a healthy charter sector.

But the story is not that clean.

This CREDO paper, which looked at charter school performance in Texas between 2011 and 2015, found a small positive effect in reading and no effect in math. Given that CREDO tracks individual students across time, and NAEP does not, the CREDO data should make us cautious in interpreting the NAEP gains as a huge victory.

Screen Shot 2018-04-24 at 9.57.03 PM

Another study found that it took Texas charters ten years to achieve the performance gains of their traditional school peers.

While it’s great to see the sector improving in Texas, perhaps with better regulation the sector wouldn’t have had to improve for ten years just to achieve neutral effects.

The story of the Texas charter sector is much more complicated than Matt’s piece indicated. The same is true for Arizona and Florida, where quasi-experimental research has found muted test score effects.

Given the different results of NAEP and CREDO, we should be trying to figure out the puzzle rather than claiming victory, as Matt tends to do.

Don’t Look at States in Isolation 

Looking at state level gains is not a great measure of whether portfolio and quarterbacks are working, as portfolio (to some extent) and quarterbacks (almost always) are city based endeavors.

For city based data, this CREDO study measured student learning effects in a bunch of cities across the country.

Washington D.C., Denver, New Orleans, and Newark all did very well. These cities are also some of the most mature portfolio cities.

Phoenix, Austin, El Paso, Forth Worth, Mesa, and San Antonio did not do well. These cities are all found in loose regulation states.

It hardly seems like a slam dunk to me that portfolio / quarterbacks are bad and loose regulation is good.

Of course, the CREDO analysis is not perfect: test scores aren’t everything and the virtual twin methodology may miss unobserved differences between students.

But looking at state based NAEP scores to make broad judgements on portfolio and quarterbacks is unwise, especially with so much other evidence available.

The portfolio / quarterback model seems to be doing some good in many cities.

Quarterbacks are a Step in the Right Direction  

Jay and Matt often criticize quarterbacks as vehicles for people who think they are smarter than everyone else (especially educators and families). I find this to be an overly simple critique.

Quarterback originated as a way to use expertise to aggregate and allocate philanthropy.

In many cities, philanthropists were funding low impact activities, often wasting it on the  pet projects of district leadership. A lot of money was spent for very little academic gain.

Quarterbacks have helped improve philanthropy: instead of just passively giving money to the district, philanthropists partner with expert management teams to try and launch and grow great non-profits.

I think this is a major improvement on the status quo. Of course, there are some drawbacks, and too much centralization of philanthropic capital poses risks. This is why I don’t think all of a city’s philanthropic capital should flow through one organization.

But quarterbacks are increasing, not decreasing, educator entrepreneurship and family choice. Yes, they do often use test score results selecting who to fund, but I suspect this will change if a better way to invest is developed over time.

In Sum

The NAEP data should not be ignored. It’s made me more open to the idea that looser regulations can lead to charter scale and quality, especially at the state level. And I found Matt’s data analysis to be quite helpful. I love it when smart people who think differently than me play with complex data sets and come to novel conclusions.

But I think there’s plenty of other state based evidence that should make us cautious, such as the CREDO Texas study.

I also think there’s a lot of evidence that the charter sectors in portfolio / quarterback cities are making a lot of gains. The NAEP data Matt and Jay site, which is state based and does not track individual students, is not convincing enough to make me deeply question our city based work.

All that being said, I look forward to reading more from Jay and Matt in the future.

I don’t take smart, critical friends for granted.

The best school district in the United States?

One of the joys of my job is the number of amazing emails that arrive in my inbox.

Below is an email (pasted with permission) from Scott Pearson, the head of the Washington DC Public Charter School Board.

On this blog, as well as on twitter, we debate a lot about regulation. We have a lot to figure out and these debates help me get smarter.

But leaders on the ground have to lead, always with imperfect information and complicated local contexts.

The DC Public Charter School Board has chosen to regulate the charter community fairly tightly on performance, but more loosely on other inputs. As Scott notes in his letter, over 40 charters have closed in Washington DC over the past decade. While I don’t know if this is right for every community, the DC charter community is providing a lot of great options for tens of thousands of children, and they have undoubtedly made DC a better city.

The continuity of the DC charter community’s success also reinforces my belief in the importance of non-profit governance. It’s hard to think of a better school district in the country, and I’m highly confident that a primary key to their success is their structure: the DC Public Charter School Board regulates and non-profits operate.

It’s a winning formula for kids.

___________________

Dear Colleagues,

We have faced nearly a year’s worth of bad news about DC Public Schools, from high teacher turnover, to faked suspension data, from inflated graduation rates, to the resignation of the DCPS chancellor, to residency fraud.  This steady drumbeat has undermined confidence in our traditional public schools – far more than is warranted in my opinion.  DCPS is a vastly better school system than it was in 2007 when Michelle Rhee and Kaya Henderson took the reins.

But I’m writing today about the other half of public schools in DC, the 120 public charter schools serving 43,340 students – nearly half (47.5%) of DC’s public school students.  Our story is mostly one of continued success, growth, popularity, and quality improvement.  I feel the need to write this because I fear that the bad news about DCPS is drowning out what continues to be a remarkable story of charter school success in our nation’s capital.

First, it’s important to say that not one of these bad news stories has been about public charter schools.  Our graduation rates check out – because for years the DC Public Charter School Board has audited the transcript of every graduating senior.  Our discipline data has never been questioned.  And there have been no allegations that anybody has jumped the lottery queue at a DC charter school.

Of course, we’re not perfect. In a sector as diverse as ours there will be unflattering news to uncover.  But at least so far none of the year’s bad news has touched DC’s charter sector.

Second, our quality keeps improving.   The NAEP “flatline” story just doesn’t apply to DC’s charter schools, as David Osborne and Emily Langhornewrite in The 74.  We were up this year in three of four grade/subject areas, with fourth-grade reading scores climbing five scale score points.  That’s more than any state in the nation.  And over ten years our growth overall is faster than any other state or district.  Charter scale scores have grown 17 points in 4th-grade math, 19 points in 4th-grade reading, and 12 points in 8th-grade math.  Each of these represents over a year’s worth of learning gains.   Only in 8th-grade reading are our scores disappointing, down 2 points over the past ten years and lagging far behind our big-city peers.

Our school leaders deserve much of the credit for this growth, but the authorizer gets some credit, too.  Since 2007 we have overseen the closure of 40 low-performing charter schools, all the while aggressively supporting growth for our highest-performing schools.

Third, our improvements are not driven by a change in our demographics.  Charter schools continue to enroll higher percentages of black and low-income students than does DC Public Schools.  And charters schools now educate the same percentage of students with disabilities as does DCPS – and higher percentages of our most disabled children.

Fourth, even as our quality improves, our schools have made remarkable progress reducing out of school suspensions and expulsions.  Out of school suspension rates are down by half since 2011-12, to under 7%.  And expulsions are down over 80% to less than 0.25% – about the national average.  We’ve achieved this through transparency and communications, not through mandates.  (Though, disappointingly, our city council is now threatening to regulate school discipline.)

Finally, demand keeps growing.  Despite the charter board adding nearly 9,000 charter school seats since 2013-14, the number of unique families on charter school waitlists has risen from 7,205 in April, 2014, to 11,317 in April, 2018.  Two-thirds of our charter schools saw their waitlist length increase from last year to this year.   Waitlists are in one sense a measure of our success because it shows families want our schools.  But is also a measure of our failure – and that of the District government – to provide our residents with enough quality schools and the facilities to house them.  We need to do better.

Thank you for reading, and please reach out if there is more information or context I can provide.

Scott Pearson

Executive Director

DC Public Charter School Board

What is California telling us about what parents want from public education?

I sit on the board of the California Charter School Association (CCSA), which is one of most effective charter associations in the nation.

Their data team put this slide together.

It’s a little complicated, but it’s very informative.

Screen Shot 2018-04-05 at 8.08.33 PM

To build the chart, CCSA looked at schools that are serving the same type of demographics and then compares their test scores.

A school that outperforms similar schools, gets a higher score (8-10 is really good); a school that underperforms similar schools, gets a lower school (1-3 is pretty bad).

This methodology – absolute test scores controlled for demographics – is imperfect, but it’s the best methodology you can use given California does not publicly report student growth scores.

Charter schools serving students in poverty outperform district schools on state tests 

43% of charter schools with higher concentrations of poverty outperform similar schools.

In California, if you are a low-income Hispanic or African-American child, you are more likely get a better education (as measured by test scores and parent demand) if you attend a charter school.

This is good news for the educators and families who are working together to create better educational outcomes in at-risk communities.

If parents are demanding schools with good test score impacts, the government’s response should be obvious: let more of these schools open.

Charter schools serving middle class students underperform district schools on state tests 

But not all charter schools are outperforming their peers on tests scores.

47% of charter schools serving middle class students perform worse than similar schools. And only 26% perform better.

So perhaps California should close some of these charter schools that serve middle class families? Research covered in this post shows that schools with negative test impacts tend not to have large positive life outcome impacts.

But here’s the odd thing: presumably, middle class families have a decent amount of information at hand when making school choices. It takes all of two minutes to scan Zillow or Great Schools to get a quick read on the absolute test score performance of any school in the state.

So why are all these relatively well resourced families sending their children to lower-performing charter schools as measured by state tests?

I’m not sure. It would be interesting to focus group and poll them to learn more.

And with regards to closure, while I surely disagree with middle class Californians on many policy issues, I’m not sure that I think I know enough about their children to close schools that have modest negative test impacts but high enrollment demand.

Charter schools don’t exist unless parents want their children to attend them 

One of the best features of charter schools is that they don’t exist unless parents choose them. No one is assigned to a charter school.

So what are we to make of this data where families in living in poverty are choosing schools with positive test scores impacts and middle class families are choosing schools with negative test score impacts?

I think the starting point should be to assume that families, on average, are in a better position to make an informed choice than government is.

Remember, government’s default assignment algorithm is to look at your family’s address and then assign your child to the nearest school. It’s not very nuanced!

My guess is that parent choice will outperform geographic assignment when it comes to finding great fits between kids and schools.

But I do think we should be open to the idea that parents, sometimes en masse, can make mistakes. And, at times, this can warrant government intervention.

Sometimes performance might be an indicator, such as when families keep sending their children to high schools with below 40% graduation rates and /or schools with extremely negative value-add scores. If less than half the kids are graduating, and those that do are barely literate, government should step in.

Sometimes lack of alignment with our nation’s professed values might be an indicator: certain public schools have at times been captured by groups, sometimes religious, that do not teach basic democratic values.

In these cases of significant performance or culture malfeasance, government should consider intervention, ideally by handing over management of the school to a non-profit organization that can achieve better results.

I don’t know enough about individual school performance to know if what’s happening in California with middle class families equates to education malfeasance, but I’m a bit skeptical.

My hunch is that once absolute test score levels surpass a certain floor (as they tend to in middle class schools), families just care a lot about other factors.

Even if I might make a different choice, I don’t know that the situation warrants government intervention.

When government should not intervene 

While it’s difficult to decide when government should intervene, it’s still pretty clear to me when government should not intervene.

When schools have both high demand and high test score impacts with students living in poverty, government should not prevent these schools from serving more students!

California should follow this common sense policy.

Unfortunately, too many school districts do not.

SaveSave

SaveSave

What do test scores tell us about schools?

Collin Hitt, Michael Q. McShane, and Patrick J. Wolf just published a report on the connection (or lack of connection) between test scores and long-term outcomes.

They looked at a bunch of school choices studies and tried to see if a school’s impact on student test scores was connected to its impact on student life outcomes.

Their conclusion: “at least for school choice programs, there is a weak relationship between impacts on test scores and later-life outcomes.”

Much of our K12 education policy is predicated on the idea that test scores are an important measure of school performance. If this is not true, behavior should change.

The Laura and John Arnold Foundation is currently funding more research on this question, and I’m eager to see what we find, especially with regards to income gains over time. This study was predicated on high school and college attainment being an indicator of long-term outcomes, but schooling isn’t always learning. So we should be careful judging school performance based on later school attainment, rather than income (or other measures).

But for now, here’s what’s on my mind after reading the study.

What do bad test score results tell us?

If I’m reading their report correctly (and I hope the authors correct me if I’m not), it seems rare that schools have a negative impact on test scores but a positive impact on long-term outcomes.

In the 126 study comparisons where test scores impacts were compared to high school or college outcomes, there were only 2 instances where a study found a significant negative impact on test scores and a significant positive impact on life outcomes.

It seems rare for a school to do really poorly on test scores but really great on life outcomes. I think the authors underemphasize this point in their paper.

What do mediocre test scores tell us?

It is much more common for schools to have a neutral (insignificant) impact on test scores but a significant positive impact on long-term outcomes. It looks like this occurred 32 times in the research review.

There may be a bunch of schools that don’t really impact test scores but are doing something that helps with long-term outcomes.

What do great test scores tell us?

There are no cases where a study found significantly positive test scores and significantly negative life outcomes.

So it seems rare for schools to jack up test scores but ruin kids lives. That’s good.

However, there were 17 instances of studies finding positive impact on test scores but neutral impacts on long-term outcomes.

So there seem to be some schools that are achieving good test results without translating these into great long-term outcomes.

How should this research affect regulation and philanthropy?

If these results hold, I think I will maintain my belief that we should replace schools with persistent very negative test scores. There appears to be little risk that these schools are really amazing schools. The negative test scores are a useful signal.

Yes, there might be other schools that are just as bad at life outcomes that are not closed because they achieve better test scores, but so long as we are closing schools that are not delivering great life outcomes, and opening schools that have a better chance of achieving great life outcomes, this seems like a worthwhile tradeoff.

But when it comes to expanding schools, if this research holds, I will rely less on positive test scores, and I think authorizers should do the same.

From an authorizer perspective, so long as a school does not have significantly negative test scores, perhaps the school should be able to expand so long as there is parent demand.

Philanthropy may also need to adjust by investing more heavily in school operators that show a positive impact on life outcomes (irregardless of test scores), and being willing to fund mediocre test score schools who either have high parent demand or who are using practices that are correlated with positive long-term outcomes (more research needed to determine what these might be).

I am very open to moving in this direction if research warrants it. The idea that it’s easier to tell a bad school than it is to identify a great school already matches my intuition, and deferring to parent judgment makes a lot of sense if we are not confident in our analysis of performance.

Charters growing in your city? You have 5 options.

Charters schools continue to scale in urban areas. In many cities, charters serve over 30% of students.

Screen Shot 2017-03-23 at 10.03.33 PM.png

In 44 cities charters serve over 20% of students.

These 44 cities, as well as many others in the future, will have to evolve their educational systems to govern a mixed portfolio of school types.

What options are available to these cities? Here’s five, some of which will be much better for children than others.

1. Implode (Detroit) 

In Detroit, the school district responded to charter growth by bankrupting itself. It lost enrollment, took on debt, and continued its academic and operational dysfunction.

In failing to respond productively to charter growth, the district hurt students and cost taxpayers nearly a billion dollars.

2. Compete (Washington D.C.)

In Washington D.C., charters now serve nearly 50% of the students. During the past decade of charter growth, the district responded by becoming perhaps the highest performing urban school district in the nation.

The district lost nearly half its students and radically increased its performance.

The district didn’t really partner with charters, it just stepped up its game.

3. Coordinate and Collaborate (Denver)

Denver Public Schools responded to charter growth by coordinating with the charter sector.

For much of the past decade, it gave charters facilities to grow in neighborhoods where more good schools were needed. The district also set-up a unified enrollment system that made it easy for families to choose easily between district and charter schools.

While there have been some bumps along the way, for the most part the district has supported the best charters to expand and has closed the worst charters.

4. Blur the Lines (Indianapolis, Camden) 

A few years ago, the Indiana legislature passed a law that allows for Innovation Schools, which are authorized by the district, have many of the autonomies of charters, are governed by non-profit boards, but still sit within the district’s enrollment and accountability reporting.

With its Renaissance Schools, Camden has done something similar: Renaissance Schools are more tightly managed by the district, but still retain most of the autonomies of charters schools.

In both Indianapolis and Camden, the district has co-opted the best of the charter model while still maintaining a tighter form of local oversight and control.

5. Govern (New Orleans)

In New Orleans, the district responded to increasing charter growth by relinquishing its operational duties and transforming into a regulator.

Rather than operate schools, the district sets performance targets, monitors for equity, and annually opens great schools and orchestrates the transformation of failing schools.

This has led to unprecedented student achievement gains.

Which Way to Go? 

While I think the last option (govern) is the best way to go, cities have also seen academic growth by competing, coordinating, and blurring the lines.

These cities are the reason I’m skeptical of people who argue that charter growth will hurt traditional public schools.

There’s an emerging group of cities who are proving this clearly doesn’t need to be the case.

I’m hopeful that their successes will be replicated much more often than not.

Why don’t we have a 10x better school?

10x

There’s a Silicon Valley mantra that your need new product needs to be 10x better than the incumbent in order for you to displace them and have a shot at a market monopoly.

Uber, for example, is a 10x product. It vastly better than taxis on so many dimensions (price, easy of use, consistency, service, etc.).

In education, it’s unclear to me that we’ve built a schools that are 10x better than the median traditional district school.

We have built schools that are 10x better than failing urban schools, and it’s no surprise that this is where the entrepreneurial sector has seen so much success.

Why haven’t we built a bunch of schools that are 10x better than an average school?

I’m not sure, but some reflections below.

1. Educators are trying to be 10x at the wrong thing 

Great tech companies usually initially succeed because their technology – not their operations – is 10x better than their competitors.

Often times, technology can be built by smallish group of highly talented people and then scaled at little marginal effort or cost.

So far, school operators have not been able to replicate this model of technological advancement and scale. This way of thinking is not in their DNA. They are still trying to squeeze 10x improvements out of areas such as program design, human resources, and operations.

It will be interesting to see if Summit, Alt School, Khan Academy and others can utilize a 100x tech backbone to scale an instructional program that, over time, evolves into a 10x better school.

2. There’s no profit motive

Perhaps. With companies like Bridge Academies, we are seeing interesting attempts at 10x breakthroughs in the for-profit international market.

On the other hand, there’s plenty of for-profit K12 and university operators in this country, and they aren’t launching 10x better schools that are displacing government and non-profit operators.

3. The education sector is over-regulated

Perhaps regulation is stifling innovation.

I’m sure that this is at least partially true, but the most in demand private schools are not very innovative. Rather, they tend to be highly selective, academically rigorous, extracurricular rich, and culturally strong.

And they also cost $30,000 a year.

So, to date, the private side of things is not exactly delivering a bunch of breakthrough innovations.

Maybe an expansion of education savings accounts will unleash some 10x products, but it’s hard to say this with great confidence.

4. The industry culture is risk averse 

Education may be attracting and retaining professionals who are generally not willing to take the risks needed to achieve 10x products.

In some sense, given that children are involved, this culture is to some extent warranted.

But maybe it needs to be loosened up a bit.

5. This is (mostly) as good as it gets

Not everything can be made better. The fork I ate my dinner with today is not that much better than a fork from the 1970s.

Perhaps this is about as good as schooling gets.

My guess? 

I don’t yet have opinions that are strong enough to warrant action beyond the work I’m already doing.

But I want to keep thinking about this.