Category Archives: Harbormasters

Response to Matt Ladner and Jay Greene

Over at Jay’s blog, Jay and Matt wrote two critical posts of portfolio management and harbormasters (our team calls them quarterbacks).

Jay and Matt think differently than I do, have different political orientations, and are sharp. Their writing makes me smarter.

I also find their post titles, writing tone, and evidence analysis to be a bit over the top. They sometimes overstate their claims and under appreciate the other side of the argument.

In these two posts, Jay and Matt use NAEP charter sector gains in Arizona, Michigan, and Texas – as well as the mediocre NAEP scores seen in Louisiana’s charter sector – to argue that portfolio management and quarterbacks aren’t working.

I found their analysis to be overly narrow. Instead of taking some new evidence in and synthesizing this with the broad set of evidence available, they anchored on to one set of data points and made too strong of claims (especially in the titles of their posts).

Don’t Look at NAEP in Isolation 

Matt is right to point out that some states with fairly loose charter regulations saw a lot of charter gains in NAEP between 2009 and 2017.

I think this should modestly increase our belief that being loose on charter openings and closings can lead, over time, to a healthy charter sector.

But the story is not that clean.

This CREDO paper, which looked at charter school performance in Texas between 2011 and 2015, found a small positive effect in reading and no effect in math. Given that CREDO tracks individual students across time, and NAEP does not, the CREDO data should make us cautious in interpreting the NAEP gains as a huge victory.

Screen Shot 2018-04-24 at 9.57.03 PM

Another study found that it took Texas charters ten years to achieve the performance gains of their traditional school peers.

While it’s great to see the sector improving in Texas, perhaps with better regulation the sector wouldn’t have had to improve for ten years just to achieve neutral effects.

The story of the Texas charter sector is much more complicated than Matt’s piece indicated. The same is true for Arizona and Florida, where quasi-experimental research has found muted test score effects.

Given the different results of NAEP and CREDO, we should be trying to figure out the puzzle rather than claiming victory, as Matt tends to do.

Don’t Look at States in Isolation 

Looking at state level gains is not a great measure of whether portfolio and quarterbacks are working, as portfolio (to some extent) and quarterbacks (almost always) are city based endeavors.

For city based data, this CREDO study measured student learning effects in a bunch of cities across the country.

Washington D.C., Denver, New Orleans, and Newark all did very well. These cities are also some of the most mature portfolio cities.

Phoenix, Austin, El Paso, Forth Worth, Mesa, and San Antonio did not do well. These cities are all found in loose regulation states.

It hardly seems like a slam dunk to me that portfolio / quarterbacks are bad and loose regulation is good.

Of course, the CREDO analysis is not perfect: test scores aren’t everything and the virtual twin methodology may miss unobserved differences between students.

But looking at state based NAEP scores to make broad judgements on portfolio and quarterbacks is unwise, especially with so much other evidence available.

The portfolio / quarterback model seems to be doing some good in many cities.

Quarterbacks are a Step in the Right Direction  

Jay and Matt often criticize quarterbacks as vehicles for people who think they are smarter than everyone else (especially educators and families). I find this to be an overly simple critique.

Quarterback originated as a way to use expertise to aggregate and allocate philanthropy.

In many cities, philanthropists were funding low impact activities, often wasting it on the  pet projects of district leadership. A lot of money was spent for very little academic gain.

Quarterbacks have helped improve philanthropy: instead of just passively giving money to the district, philanthropists partner with expert management teams to try and launch and grow great non-profits.

I think this is a major improvement on the status quo. Of course, there are some drawbacks, and too much centralization of philanthropic capital poses risks. This is why I don’t think all of a city’s philanthropic capital should flow through one organization.

But quarterbacks are increasing, not decreasing, educator entrepreneurship and family choice. Yes, they do often use test score results selecting who to fund, but I suspect this will change if a better way to invest is developed over time.

In Sum

The NAEP data should not be ignored. It’s made me more open to the idea that looser regulations can lead to charter scale and quality, especially at the state level. And I found Matt’s data analysis to be quite helpful. I love it when smart people who think differently than me play with complex data sets and come to novel conclusions.

But I think there’s plenty of other state based evidence that should make us cautious, such as the CREDO Texas study.

I also think there’s a lot of evidence that the charter sectors in portfolio / quarterback cities are making a lot of gains. The NAEP data Matt and Jay site, which is state based and does not track individual students, is not convincing enough to make me deeply question our city based work.

All that being said, I look forward to reading more from Jay and Matt in the future.

I don’t take smart, critical friends for granted.

Applying portfolio reforms to postsecondary problems

I just got back from a trip to New Orleans, which continues to be a well of friendship and inspiration.

I. Where should you spend the next philanthropic dollar? 

In a few conversations, the following questions came up:

  1. Are the kids we serve going to succeed in life after high school? What will their lives be like when they are 30? Will they be living meaningful and happy lives?
  2. Is the marginal dollar of philanthropy best spent on making the K-12 system better (after 10 years of improvements) or trying to overhaul the post-secondary landscape?
  3. If you wanted to radically improve post-secondary, what would you do?

II. Post-Secondy portfolio 

The K-12 portfolio mindset entails viewing an educational system in terms of operators (running schools) and seats (how many students are served).

This mindset could also be applied to post-secondary.

By 2020 or so, New Orleans will be graduating around 3,000 students a year.

Let’s say that about 1,500 of them will be prepared to succeed in a four year college; 1,000 of them will be prepared to succeed in a 1 to 2 year credentialing program; and 500 of them will need deep support to enter the workforce and exit crisis situations.

Of the four year college students, you might need 500 to 1,000 “KIPP to College” type supports to ensure students make it through.

For the credentialing programs, you’d need 1,000 seats that can reliably produce students with employable credentials.

For the crisis students, you’d need employment and social service operators that could transition students into jobs.

III. Post-Secondary investment intermediaries 

Instead of assuming this will naturally happen in New Orleans (or any other city), you could capitalize a new or existing non-profit intermediary to launch, recruit, and support post-secondary providers.

At the outset, the intermediary would create a business plan where it laid out how money it would need to get X% coverage on the aforementioned 3,000 seats.

High-Quality existing local providers (like the coding bootcamp Operation Spark) could cover some of the seats, and national providers like Match Beyond could be recruited in.

Overtime, you’d expand what was working, close what wasn’t, and support new entrepreneurs to keep innovation going.

IV. Getting funding streams right

Most states subsidize mediocre public universities; the federal government tops this off with Pell grants.

To make the 3,000 seat post-secondary strategy viable, you’d need to blend a mixture of public support and tuition to make providers sustainable.

Louisiana’s course choice provides a revenue stream for programs that started working with kids while they’re in high school.

Creating a new university that housed many of these programs could allow for the accessing of Pell grants.

Wage contigent loan programs could also be an option for programs that were consistently placing graduates in high-performing jobs.

V. Who are the entrepreneurs that will seek out the 10x play?

The early New Orleans K12 entrepreneurs felt that they could deliver something to students that was significantly better than the existing system.

They were right.

A post-secondary transformation won’t happen on its own.

It will take a set of entrepreneurs to put forth a plan, galvanize funding, and spend a decade building the new system.

Is this the right play? If so, who will step up?

5 Key Metrics Cities Should Use to Drive Educational Improvement

harbormaster

Andreesen Horowitz recently published a list of 16 metrics that venture capitalists often look at when making investment decisions.

This made me reflect on the state of metrics for city based education reform.

Over the past year, I’ve been thinking a lot about what metrics city based leaders (either as a formal harbormaster or as a coordinated effort philanthropists and reformers) should use to set goals and track their progress.

To be blunt, I think many of the goals that city leaders set are garbage.

I’m highly skeptical about setting goals such as: “by 2025 80% of our 3rd graders will be reading on grade level.”

This is generally what you see in district strategic plans or in city based collective action plans. While I don’t have exact numbers, my guess is that approximately 100% of districts fail to meet these type of goals.

The reason is simple: these goals are not tethered to a strategy that has a realistic chance of producing the desired results.

So what should city leaders do? Here’s some metric that I find useful:

#1 New High-Quality Seat Creation

Cities should set hard targets on how many new high-quality seats (students served) they will create via new school openings (charter, traditional, or private).

The total seat number should at least be 2% of the total student population for larger cities, and closer to 5% for smaller cities. So a city with 500,000 students should be opening ~10,000 new seats a year, and a city with 40,000 students should be opening ~2,000 new seats a year.

The total seat number should also be discounted in that all new seat will be high-quality. For mature cities, I’d estimate a 65-75% high-quality rate. For cities just getting new school development off the ground, I’d estimate closer to 50-65%.

#2 Medium-Quality Seat Improvement

Assuming a city is making sound investments in talent, and there is a well structured accountability system, it is likely that medium-quality schools will improve over time. I don’t really know how to set this target, as I haven’t seen good data analysis on what we can expect from improvement. If I had to pick a number, I’d shoot for 1-2% of medium quality seats become high-quality seats each year.

#3 Teacher Pipeline

One of the biggest mistakes we made in New Orleans was opening schools faster than we were building out high-quality teacher pipelines. City leaders should be building teacher pipelines that meet the needs of their expanding high-quality school sector.

Ideally, I’d like to see 50% coverage of the teacher pipelines needed to meet the needs of all high and medium quality schools. When I talk to school leaders, they estimate that 50-70% of their new hires are new teachers, so I don’t think you need 100% coverage of the new hire needs with entry teacher pipelines.

In other words, if you have 10,000 high-quality and medium-quality seats, and you have a 20:1 student teacher ration, and you estimate 20% turnover, then you need 100 teachers a year for these schools. 50% of this number would mean that you want 50 teachers a year coming from high-quality pipelines.

#4 Parent Voice

This is another area why I’m not exactly sure what the number should be. But my rough guess is that high-quality schools should be able to support at least 5% of their families to speak in support of high-quality schools at public meetings. So if a city has 20,000 students in high-quality schools, and we assume this represents 15,000 families, then at least 750 families should participate in public forums that impact high-quality schools.

#5 Community Support

I think cities should attempt to have at least 50% of citizens answering affirmative to the question: “is education improving in our city?”

____

I’m still mulling much of this over. I’m very sure the above is not exactly right.

But I do think the above metrics represent a strategic world view that has a high chance of increasing educational opportunity.

Moreover, these metrics can be rolled up to make projections on outcomes such as high school graduation rates, but I do think the real strategic and analytical firepower is getting the short-term execution based metrics right.

Overtime, I’d love to see us develop more precision about what ambitious but realistic targets are across all these areas.

Lastly, if you have other ideas for metrics, I’d love to hear them.

Isaac Asimov, Harbormasters, and Charter Sectors

asimove

I’m reading Isaac Asimov’s Foundation Series. The series, at least from what I gather from the first part of the book, is based on one of the main character’s study of “psychohistory,” which involves combining politics, sociology, history, economics, and psychology into a mathematical science that can be used to predict (and manipulate) future geopolitical events.

What I’ve enjoyed most about the novel is how psychohistory predicts the efficacy of different solutions based on different environmental conditions. In one case, a multi-kingdom alliance solves the problem; in another, forming a religion averts the crisis. Great psychohistorians can see centuries of chess moves ahead: they aim to tailor solutions to problems that were a natural outgrowth of the last set of solutions.

In short, the book does an excellent job of demonstrating how strategy must fit environment.

Again, I’m only two hundred pages in the first book, so the series might go in a totally different direction. Please don’t include plot spoilers in the comments.

___

Last week, I was on a call with a colleague and we were discussing what would be needed to double the rate of high-quality charter growth in the country. At first, we discussed broad, national solutions. Very quickly (and smartly) he then pivoted the conversation by saying: what if we looked at a sample of ten charter sectors across the country?

So we quickly ran through a list of cities and discussed what it would take to double the high-quality charter sector in each of the cities.

While there were many commonalities, there were large differences in optimal strategies between different cities.

___

At New Schools for New Orleans, we went through (at least) three major strategic shifts in charter school investment. From 2005-2009, we invested a lot in new operator creation. From 2011 to 2014, we invested a lot in growing networks. Currently, the organization is trying to build capacity in  the charter sector to solve incredibly difficult challenges, such as how to best serve students with complex special needs.

This is not to say that the aforementioned strategies were the only types of investing done in the periods; only that in each phase priorities shifted based on sector conditions.

___

Education Cities is an organization that supports New Schools for New Orleans type entities across the country.

The describe their work as the following:

Our members serve as education “harbormasters” with deep ties to their communities. Like maritime harbormasters, who facilitate safe and cooperative navigation in a challenging space, education harbormasters build and coordinate the efforts to improve education in their city. Together, our members – nonprofits, foundations, and civic organizations – are improving opportunities for millions of children and their families.

Whether or not you like the phrase “harbormaster” – the goal is, I think, a sound one: supporting local organizations that can drive change based on local conditions.

___

Unfortunately, the charter sector does not have any psychohistorians amongst it ranks. As such, it’s difficult to predict how the next fifty years of high-quality charter school growth will (or will not) occur.

I do think that a rough taxonomy of charter sector maturation will form over time. We’ll continue to build knowledge about what works during different stages of sector development.

But most of this taxonomy will be backward looking, not forward looking. And it will never be perfectly tailored to any individual city’s conditions.

This is why I view well run harbormasters to be of use. By fitting strategy to environment, they can come up with novel solutions to the most pressing local solutions – taking the best from what’s been tried nationally, but always with an eye toward local conditions.

This is something that national foundations, national education organizations, and federal and state government entities will always struggle to do.

The goal of education reform should not be finding and adopting current best practices.

The goal of education reform should be to build learning ecosystems that constantly evolve.

Harbormasters can be a key part of these ecosystems.