Category Archives: Portfolio

Response to Matt Ladner and Jay Greene

Over at Jay’s blog, Jay and Matt wrote two critical posts of portfolio management and harbormasters (our team calls them quarterbacks).

Jay and Matt think differently than I do, have different political orientations, and are sharp. Their writing makes me smarter.

I also find their post titles, writing tone, and evidence analysis to be a bit over the top. They sometimes overstate their claims and under appreciate the other side of the argument.

In these two posts, Jay and Matt use NAEP charter sector gains in Arizona, Michigan, and Texas – as well as the mediocre NAEP scores seen in Louisiana’s charter sector – to argue that portfolio management and quarterbacks aren’t working.

I found their analysis to be overly narrow. Instead of taking some new evidence in and synthesizing this with the broad set of evidence available, they anchored on to one set of data points and made too strong of claims (especially in the titles of their posts).

Don’t Look at NAEP in Isolation 

Matt is right to point out that some states with fairly loose charter regulations saw a lot of charter gains in NAEP between 2009 and 2017.

I think this should modestly increase our belief that being loose on charter openings and closings can lead, over time, to a healthy charter sector.

But the story is not that clean.

This CREDO paper, which looked at charter school performance in Texas between 2011 and 2015, found a small positive effect in reading and no effect in math. Given that CREDO tracks individual students across time, and NAEP does not, the CREDO data should make us cautious in interpreting the NAEP gains as a huge victory.

Screen Shot 2018-04-24 at 9.57.03 PM

Another study found that it took Texas charters ten years to achieve the performance gains of their traditional school peers.

While it’s great to see the sector improving in Texas, perhaps with better regulation the sector wouldn’t have had to improve for ten years just to achieve neutral effects.

The story of the Texas charter sector is much more complicated than Matt’s piece indicated. The same is true for Arizona and Florida, where quasi-experimental research has found muted test score effects.

Given the different results of NAEP and CREDO, we should be trying to figure out the puzzle rather than claiming victory, as Matt tends to do.

Don’t Look at States in Isolation 

Looking at state level gains is not a great measure of whether portfolio and quarterbacks are working, as portfolio (to some extent) and quarterbacks (almost always) are city based endeavors.

For city based data, this CREDO study measured student learning effects in a bunch of cities across the country.

Washington D.C., Denver, New Orleans, and Newark all did very well. These cities are also some of the most mature portfolio cities.

Phoenix, Austin, El Paso, Forth Worth, Mesa, and San Antonio did not do well. These cities are all found in loose regulation states.

It hardly seems like a slam dunk to me that portfolio / quarterbacks are bad and loose regulation is good.

Of course, the CREDO analysis is not perfect: test scores aren’t everything and the virtual twin methodology may miss unobserved differences between students.

But looking at state based NAEP scores to make broad judgements on portfolio and quarterbacks is unwise, especially with so much other evidence available.

The portfolio / quarterback model seems to be doing some good in many cities.

Quarterbacks are a Step in the Right Direction  

Jay and Matt often criticize quarterbacks as vehicles for people who think they are smarter than everyone else (especially educators and families). I find this to be an overly simple critique.

Quarterback originated as a way to use expertise to aggregate and allocate philanthropy.

In many cities, philanthropists were funding low impact activities, often wasting it on the  pet projects of district leadership. A lot of money was spent for very little academic gain.

Quarterbacks have helped improve philanthropy: instead of just passively giving money to the district, philanthropists partner with expert management teams to try and launch and grow great non-profits.

I think this is a major improvement on the status quo. Of course, there are some drawbacks, and too much centralization of philanthropic capital poses risks. This is why I don’t think all of a city’s philanthropic capital should flow through one organization.

But quarterbacks are increasing, not decreasing, educator entrepreneurship and family choice. Yes, they do often use test score results selecting who to fund, but I suspect this will change if a better way to invest is developed over time.

In Sum

The NAEP data should not be ignored. It’s made me more open to the idea that looser regulations can lead to charter scale and quality, especially at the state level. And I found Matt’s data analysis to be quite helpful. I love it when smart people who think differently than me play with complex data sets and come to novel conclusions.

But I think there’s plenty of other state based evidence that should make us cautious, such as the CREDO Texas study.

I also think there’s a lot of evidence that the charter sectors in portfolio / quarterback cities are making a lot of gains. The NAEP data Matt and Jay site, which is state based and does not track individual students, is not convincing enough to make me deeply question our city based work.

All that being said, I look forward to reading more from Jay and Matt in the future.

I don’t take smart, critical friends for granted.

What do test scores tell us about schools?

Collin Hitt, Michael Q. McShane, and Patrick J. Wolf just published a report on the connection (or lack of connection) between test scores and long-term outcomes.

They looked at a bunch of school choices studies and tried to see if a school’s impact on student test scores was connected to its impact on student life outcomes.

Their conclusion: “at least for school choice programs, there is a weak relationship between impacts on test scores and later-life outcomes.”

Much of our K12 education policy is predicated on the idea that test scores are an important measure of school performance. If this is not true, behavior should change.

The Laura and John Arnold Foundation is currently funding more research on this question, and I’m eager to see what we find, especially with regards to income gains over time. This study was predicated on high school and college attainment being an indicator of long-term outcomes, but schooling isn’t always learning. So we should be careful judging school performance based on later school attainment, rather than income (or other measures).

But for now, here’s what’s on my mind after reading the study.

What do bad test score results tell us?

If I’m reading their report correctly (and I hope the authors correct me if I’m not), it seems rare that schools have a negative impact on test scores but a positive impact on long-term outcomes.

In the 126 study comparisons where test scores impacts were compared to high school or college outcomes, there were only 2 instances where a study found a significant negative impact on test scores and a significant positive impact on life outcomes.

It seems rare for a school to do really poorly on test scores but really great on life outcomes. I think the authors underemphasize this point in their paper.

What do mediocre test scores tell us?

It is much more common for schools to have a neutral (insignificant) impact on test scores but a significant positive impact on long-term outcomes. It looks like this occurred 32 times in the research review.

There may be a bunch of schools that don’t really impact test scores but are doing something that helps with long-term outcomes.

What do great test scores tell us?

There are no cases where a study found significantly positive test scores and significantly negative life outcomes.

So it seems rare for schools to jack up test scores but ruin kids lives. That’s good.

However, there were 17 instances of studies finding positive impact on test scores but neutral impacts on long-term outcomes.

So there seem to be some schools that are achieving good test results without translating these into great long-term outcomes.

How should this research affect regulation and philanthropy?

If these results hold, I think I will maintain my belief that we should replace schools with persistent very negative test scores. There appears to be little risk that these schools are really amazing schools. The negative test scores are a useful signal.

Yes, there might be other schools that are just as bad at life outcomes that are not closed because they achieve better test scores, but so long as we are closing schools that are not delivering great life outcomes, and opening schools that have a better chance of achieving great life outcomes, this seems like a worthwhile tradeoff.

But when it comes to expanding schools, if this research holds, I will rely less on positive test scores, and I think authorizers should do the same.

From an authorizer perspective, so long as a school does not have significantly negative test scores, perhaps the school should be able to expand so long as there is parent demand.

Philanthropy may also need to adjust by investing more heavily in school operators that show a positive impact on life outcomes (irregardless of test scores), and being willing to fund mediocre test score schools who either have high parent demand or who are using practices that are correlated with positive long-term outcomes (more research needed to determine what these might be).

I am very open to moving in this direction if research warrants it. The idea that it’s easier to tell a bad school than it is to identify a great school already matches my intuition, and deferring to parent judgment makes a lot of sense if we are not confident in our analysis of performance.

What will be normal for superintendents in 2028?

Education change happens slowly until it happens everywhere. And then a new normal exists.

Right now, it is normal for superintendents to directly control all the public schools in their city. It also normal for them to fight off educators (either within their system or external charter operators) who threaten this control.

But this is changing. A new wave of superintendents are looking to charter operators to help them solve their toughest problems. And they’re not doing so in light touch ways, like best practice sharing. Instead, they’re handing over control of some of their most struggling schools to charter schools.

Atlanta 

In Atlanta, the superintendent is partnering with the city’s best charter operators to operate underperforming schools in the city.

Here is how one Atlanta high school student responded to her plan:

Omari Hargrove, a junior at Carver School of the Arts, isn’t looking for guarantees.

At a recent community meeting, Carver students spoke about daily fights on campus, gambling at school and classmates more focused on selling drugs, than studying.

“I’m not sure if the changes they’re introducing are the ones they need, but the situation we have now is not acceptable,” Hargrove said.

“Maybe they’ll work,” he added. “At least it’s a plan.”

San Antonio 

The superintendent in San Antonio is taking a similar approach. Texas recently passed a law, Senate Bill 1882, which encourages both superintendents and charter operators to work together to turnaround failing schools.

Denver

In Denver, the district partnered with a charter operator, University Prep, to turnaround a struggling school. A year later, the school posted the highest math growth in the state of Colorado.

Indianapolis 

In Indianaplis, the district utilized the state’s Innovation law to partner with charter operators, and the first wave of schools saw an increase in test scores. One of those schools, while managed by a charter operator (Phalen Academies), was actually led by a group of district leaders who wanted more freedom (Project Restore).

Memphis

Over the past few years, the state, frustrated with local inaction, has forced charter partnerships in Memphis. But last week, the superintendent told the local paper he was willing to lead the partnership effort himself. A locally elected board member also indicated willingness: “As we look at what’s best for kids, we have to look at all options,” she said.

What will be normal in 2028?

That’s hard to tell. But I hope that what we’re seeing from this courageous superintendents becomes the new normal.

A superintendent’s job is to get the highest number of kids in the best schools as quickly as possible.

That’s a job that requires all hands on deck.

Any group of educators who can make things better quickly should be given that opportunity, charter schools included.

Over the coming decade, I hope what we see in Atlanta, San Antonio, Denver, Indianapolis, and Memphis is the new normal.

Could Newark have achieved more?

Harvard researchers just published a study on Newark.

In large part due to Mark Zuckerberg’s $100 million gift to the reform efforts, the city has seen a lot of coverage, including the well done book The Prize.

Most of this coverage has been negative.

The Harvard Study: Teasing Out Annual Effects From 2010-2016

The Harvard study found that the Newark reforms, in the most recent year of the study, had a positive impact on ELA (.07 SD) and no impact on math. In the early years of the reforms, the achievement effects were negative.

Unfortunately, the researchers only looked at annual effects (were there positive or negative effects in any given year) rather than calculating cumulative impacts (what was the total impact over these years). Because of this, we can’t perfectly compare the Newark results with results of other city based studies, such as Doug Harris’ evaluations of the New Orleans efforts.

What Caused the Positive Effects? 

While the authors didn’t calculate the cumulative effects of the reforms, they did do something wonderful.

The authors separated out the effects of two different strategies: (1) improving existing schools vs. (2) expanding high-performing schools, closing low-performing schools, and facilitating the transfer of students out of low-performing schools and into high-performing schools.

The improving existing schools approach included replacing large numbers of principals, renegotiating the union contract, implementing new data systems, and extending learning time.

The open / close / shift enrollment approach included adding to the enrollment of high-performing charter and district schools, closing underperforming schools (11 traditional schools and 3 charters), and implementing a unified enrollment system that made choosing schools much easier for families. These reforms increased charter enrollment from 14% to 28%.

The results are striking.

In math, the improve strategy achieved a .08 negative effect in math while the open / close / shift strategy had a positive .04 effect.

In ELA both strategies had positive effects, but the open / close / shift was responsible for 62% of the overall positive effect.

Perhaps most importantly: the open / close / shift strategy achieved positive effects in every year of the study. 

Opening and closing schools, and shifting student enrollment, increased student learning.

Could Newark have Achieved New Orleans’ Level Gains?

In roughly the same number of years as the Newark reforms, New Orleans achieved positive .2-.4 standard deviation effects.

New Orleans, of course, relied much more on the open / close / shift than it did on improvement.

As noted above, we can’t do a direct apples-to-apples comparison of the studies. But I’d be surprised if Newark’s cumulative effect is as large as the New Orleans effect given that Newark saw negative effects in 3 of the 5 years of reforms.

We’ll never know if Newark could have achieved similar gains to New Orleans, but in many ways the Newark circumstances were more enviable than those of New Orleans.

Newark is home to one of the highest-performing charter sectors in the country, while New Orleans had no large effective charter operators in the city right after Katrina.

Newark’s initial philanthropic investment was much larger than that received in New Orleans after Katrina. And the public per-pupil in Newark is about 3x that of New Orleans.

Newark students also didn’t have to go through the immense trauma of Katrina.

That being said: perhaps New Orleans style reforms would have not worked in Newark.

Maybe the politics would not have allowed for it… though it’s worth remembering that Shavar Jeffries, now the CEO of Democrats for Education Reform, was only about 2,000 votes away from beating Ras Baraka in the 2014 Newark mayoral race.

Or perhaps the charter sector would not have been able to grow much faster than it did; notably, charter effects did decline as they enrolled more students… though perhaps more charter capacity would have emerged under better conditions.

All told, my guess is that Newark did not increase educational opportunity as much as it could have.

A 2010 Prediction 

On this blog, I try to be honest about both my mistakes and successes.

It would be disingenuous for me to write a post arguing that, in 2010, no one could have predicted which strategy would have been better.

The fact is that I did predict this. In 2010, when it became clear that there was going to be major reforms in Newark, I created a powerpoint deck to give advice to national educational leaders. In the deck, I modeled out how quickly I thought you could expand great schools and close failing schools.

Given the relatively small size of Newark public schools (50,000 students), as well as the number of high-quality charter schools, I felt confident that the vast majority of Newark students could attend a high-quality school within ten years.

This ended up being the strategy we took in New Orleans.

It was not the strategy taken in Newark.

Learning 

I have a lot of love for the philanthropists, political leaders, and educators who spent the good part of a decade trying to make things better for kids. I know many of them well and have had nothing but positive interactions.

In terms of rigorous third party research results, we also knew a lot less in 2010 than we do now. There have been a wave of recent studies documenting the positive effects of expanding the best schools and closing the worst.

And while we still suffer from a dearth of research on citywide turnaround efforts, outside of the early results in Lawrence, Massachusetts, we really haven’t seen a citywide turnaround reform effort significantly raise achievement without a focus on opening and closing schools (research has shown that Washington D.C. improved their public schools, but the city also leaned heavily on opening and closing schools).

So perhaps the better question is not what should have Newark leaders done in 2010, but what should similarly situated civic leaders do in 2017?

Civic leaders should embrace strategies that maximize the expansion of the best schools and close the worst schools.

Of course this need not be the only focus of a city turnaround effort, but the Newark study has increased my belief that it should be the primary focus.

CREDO’s school closure research validates portfolio and golden tickets

 

CREDO just came out with a study on school closures. Matt Barnum gives a good write up in Chalkbeat (and continues to far surpass NYT and WAPO in his analysis of complicated research).

Achievement increases when you close low performing schools and students transfer to better schools 

Overall, the research increased my belief in the idea that great schools should expand and failing schools should be closed or transformed (the basis of the portfolio model).

My only reservation with this study is that it defined low-performing schools by absolute performance rather than growth in achievement; however, student achievement still grew when students moved from lower to higher absolute performing schools, so perhaps many of the low absolute schools were low growth schools as well.

The research adds to the body of evidence that shows: if you…

(1) Close lower performing schools;

(2) Increase the number of high-performing schools;

(3) Ensure students from the closed schools get into the higher-performing schools; then

(4) Educational opportunity will increase.

See below from CREDO on the student achievement effect of a student transferring from a closed school to a superior school.

Screen Shot 2017-08-27 at 9.12.18 AM

Screen Shot 2017-08-27 at 9.12.29 AM

One other thing: this type of analysis doesn’t capture the positive effect of the failing school no longer existing; a city implementing these strategies should see additional gains from no new students ever having to attend the failing school.

“Golden ticket” policies can ensure students attending closed schools get into better schools

Cities that use unified enrollment systems can easily guarantee that students leaving closed schools have access to better schools by a “golden ticket” policy. This policy gives students exiting closed schools first access to any open seats in high-quality schools in the city.

While I find the policy name “golden ticket” to be crass, it’s a policy that can do wonders for educational equity.

Too often students in failing schools are shuffled from one underperforming school to another; a golden ticket policy can prevent this.

Should we support increased school closures in majority white communities?

Numerous commentators pointed out that schools with +80% minority students were more likely to be closed than schools that had lower minority enrollment.

Most took this as a sign of inequitable treatment toward minority communities.

However, a growing body of research indicates that school closure increases educational opportunity so long as student have access to better schools. And although I wish this wasn’t the case, my hunch is that majority white communities likely have a higher concentration of better schools than your average minority community: this means that school closures are likely to be even more effective in majority white communities.

My guess is a lack of closures in majority white communities is leading to reduced educational opportunity.

School closure is incredibly hard

While I believe that thoughtfully implemented school closure policies will benefit children, I know that closing schools is hard for students, families, educators, and politicians.

But sometimes doing the hard thing can help students.

And a growing body of evidence is pointing to the idea that a well implemented school closures is one of those hard things that can ultimately make things better for students and communities.

__

Lastly: thinking of everyone in Houston right now. Really hoping that friends, colleagues, and everyone else in city are ok.

SaveSave

SaveSave

Phase II of the charter and choice research agenda is extremely important

In a recent post, I summarized much of the research on charter school and high choice cities.

The short of it: there’s a strong body of evidence that urban charter schools outperform traditional schools, and a nascent body of evidence that thoughtfully implemented high choice cities can deliver strong achievement gains.

It’s worth taking a moment to celebrate this: the research on urban charter schools is an impressive body of work built on twenty years of studying the incredible efforts of educational entrepreneurs. It’s wonderful to see these schools working.

___

But as charters scale, additional questions need to be studied.

More specifically:

  1. Do urban charter school students achieve better long-term life outcomes?
  2. What happens to traditional schools when charter schools expand?
  3. What happens to students when under-enrolled traditional schools eventually close?
  4. What happens to cities that transition to majority charter systems?

A few recent studies have shown that charter school students achieve better long-term life outcomesthat charter school expansion can improve traditional schools; that student attending schools that are closed actually benefit from the closure so long as good new schools are being continually opened; and, as noted in the previous post, that majority charter cities can deliver strong achievement gains.

But there are also studies that indicate the opposite. Charter students in Texas did not have greatly improved life outcomes; students in Baton Rouge did not increase achievement after their schools were closed; and cities such as Detroit continue to struggle despite high charter penetration.

___

Personally, I most care about questions (1) and (4). If charter schools end up delivering better life outcomes, and majority charter cities work, I’d be willing to accept some interim negative effects of charter expansion and school closures. But I understand that the latter effects could cause hardship for families, and I’m heartened to see early examples that show that charter expansion can be coupled with gains for all students.

So here’s to holding out for the best scenario: I truly hope that phase II of the research agenda shows that charters increase life outcomes, that charter school expansion and failing school closure benefits all students, and that majority charter systems deliver benefits for all students.

Initial research indicates that this might be possible, which is really exciting.

Applying portfolio reforms to postsecondary problems

I just got back from a trip to New Orleans, which continues to be a well of friendship and inspiration.

I. Where should you spend the next philanthropic dollar? 

In a few conversations, the following questions came up:

  1. Are the kids we serve going to succeed in life after high school? What will their lives be like when they are 30? Will they be living meaningful and happy lives?
  2. Is the marginal dollar of philanthropy best spent on making the K-12 system better (after 10 years of improvements) or trying to overhaul the post-secondary landscape?
  3. If you wanted to radically improve post-secondary, what would you do?

II. Post-Secondy portfolio 

The K-12 portfolio mindset entails viewing an educational system in terms of operators (running schools) and seats (how many students are served).

This mindset could also be applied to post-secondary.

By 2020 or so, New Orleans will be graduating around 3,000 students a year.

Let’s say that about 1,500 of them will be prepared to succeed in a four year college; 1,000 of them will be prepared to succeed in a 1 to 2 year credentialing program; and 500 of them will need deep support to enter the workforce and exit crisis situations.

Of the four year college students, you might need 500 to 1,000 “KIPP to College” type supports to ensure students make it through.

For the credentialing programs, you’d need 1,000 seats that can reliably produce students with employable credentials.

For the crisis students, you’d need employment and social service operators that could transition students into jobs.

III. Post-Secondary investment intermediaries 

Instead of assuming this will naturally happen in New Orleans (or any other city), you could capitalize a new or existing non-profit intermediary to launch, recruit, and support post-secondary providers.

At the outset, the intermediary would create a business plan where it laid out how money it would need to get X% coverage on the aforementioned 3,000 seats.

High-Quality existing local providers (like the coding bootcamp Operation Spark) could cover some of the seats, and national providers like Match Beyond could be recruited in.

Overtime, you’d expand what was working, close what wasn’t, and support new entrepreneurs to keep innovation going.

IV. Getting funding streams right

Most states subsidize mediocre public universities; the federal government tops this off with Pell grants.

To make the 3,000 seat post-secondary strategy viable, you’d need to blend a mixture of public support and tuition to make providers sustainable.

Louisiana’s course choice provides a revenue stream for programs that started working with kids while they’re in high school.

Creating a new university that housed many of these programs could allow for the accessing of Pell grants.

Wage contigent loan programs could also be an option for programs that were consistently placing graduates in high-performing jobs.

V. Who are the entrepreneurs that will seek out the 10x play?

The early New Orleans K12 entrepreneurs felt that they could deliver something to students that was significantly better than the existing system.

They were right.

A post-secondary transformation won’t happen on its own.

It will take a set of entrepreneurs to put forth a plan, galvanize funding, and spend a decade building the new system.

Is this the right play? If so, who will step up?