What can we learn from the most driven and knowledgeable public school parents?

This is the second blog post in a row about learning from experience and observation rather than research.

I sometimes anchor a bit too much on research and am trying to balance this mindset with a stronger observational approach.

___

One common critique of charter schools is that they draw the most driven knowledgeable families away from the traditional public school system.

According to this argument, we should be a bit suspicious of charter school test score gains, and we should be very worried that the remaining students left in the traditional system will include very high concentrations of at-risk students.

I think there is some merit to the first critique of inflated test score effects, but there’s enough positive results from randomized controlled experiments of high-performing charter schools that I’m not too worried about it.

I do worry a lot about concentrating the hardest to reach students in a smaller and smaller number of traditional schools. This definitely occurred in New Orleans, where the last 20% of charter students were much harder to serve than the first 20%.

My hope is that this issue can be somewhat ameliorated by cities directly replacing their most underperforming schools and using unified enrollment systems to make finding a better school easier for all families.

___

But there is something odd about critiquing charter schools based on the fact that they attract the most driven and knowledgeable parents.

Charter detractors are basically admitting that the more parents know the more likely they are to want to attend a charter school.

This is a strong argument for charter schools.

It would be useful to better understand why the most driven and knowledgeable parents are so drawn to charter schools. Is it because of increased test scores? A hope for better peers effects? Are they simply running away from something that is not working for their kid?

I’m not sure. The emerging research coming out of unified enrollment systems is helpful on this front, but it’s all correlational, so we should be cautious.

But the fact that the most driven and knowledgeable parents are seeking out charter schools increases my belief that charter schools are a positive force for families.

Sometimes simply watching what people do is a great way to learn.

What can we learn from school board meetings?

In forming opinions on policy, it’s good to balance research and lived experience.

Research is an invaluable tool in helping us test our beliefs, but the most rigorous research is limited to what can be measured, and not everything can be measured.

When it comes to public school policy, attending school board meetings has been a major part of my lived experience.

Over the past decade, I’ve attended two different types of school board meetings: elected school board meetings and non-profit charter school board meetings.

 

The elected school board meetings I’ve attended  range from dysfunctional (school board members screaming at each other) to misguided (school board members discussing random topics that have little to do with governing public schools) to sometimes useful (reasoned debate on systems level policy).

Unfortunately, I’ve attended many more dysfunctional and misguided elected school board meetings than I’ve attended useful elected school board meetings.

The non-profit charter school board meetings tend to be much more productive. More often than not, the board members are focused and knowledgeable. This is especially true of larger non-profit charter networks, where the board has been around for over a decade and governed through organization scaling.

Yes, I’ve attended a few dysfunctional charter school board meetings, but they are not the norm.

This lived experience in attending board meetings has shaped my policy views. Based on a decade of attending board meetings, I believe that non-profit boards will be better at governing schools than elected boards will be.

I do believe we need democratic oversight over our public schools. Elections allow us to debate values, and these values should shape how we oversee our public schools.This oversight can come in a variety of forms, from elected boards to mayoral control. The New Orleans elected school board, for example, oversees a nearly 100% non-profit charter schools system.

But while we need elections to debate values, we don’t need elections to govern individual or networks of schools.

My lived experience leads me to believe that this duty is best held by non-profit boards.

Most students in NOLA whose school is closed end up in one of their top choices for the following year. Here’s how.

This post from Ed Navigator is worth reading. It covers schools closures in New Orleans.

Over ten years after Katrina, and under an elected school board, New Orleans continues to selectively close underperforming schools.

I view this as a good thing, given the growing body of research that shows that school closures help kids when the students end up in better schools.

New Orleans uses a unified enrollment system to help kids get into better schools.

The unified enrollment system gives preference to students whose schools were closed the year before. If your school was closed, the algorithm bumps you to the top of the list for any school you want to get into.

Ed Navigator works with families whose schools have been closed, so that they can help select great schools.

The result?

This year, 87% of students who attended a closing school and used the enrollment system received on of their top three choices for the next school year.

94% of the students will now attend a school that is rated higher by the state’s grading system.

The system is by no means perfect. My biggest critique is that the state’s grading system still relies too heavily on absolute test scores (rather than growth). I also understand the counterarguments that government should never close schools and should instead let enrollment patterns (driven by parental choice) determine which schools grow and which close.

But I would rather have the New Orleans enrollment and closure system than just about any other big city system in the country.  In too many cities, really bad schools stay open for too long. And if anything happens to them, kids often end up in schools that are just as bad.

This is not what happens in New Orleans.

It’s also great to see Parag Pathak (and his colleagues) work in action. Parag recently won the John Bates Clark award in part because of his contributions to working on unified enrollment systems.

It’s rare that an idea goes from the ivory tower to think tanks to actual implementation by a democratically elected body to  helping citizens.

This is really great to see. And really great for kids in New Orleans.

Response to Matt Ladner and Jay Greene

Over at Jay’s blog, Jay and Matt wrote two critical posts of portfolio management and harbormasters (our team calls them quarterbacks).

Jay and Matt think differently than I do, have different political orientations, and are sharp. Their writing makes me smarter.

I also find their post titles, writing tone, and evidence analysis to be a bit over the top. They sometimes overstate their claims and under appreciate the other side of the argument.

In these two posts, Jay and Matt use NAEP charter sector gains in Arizona, Michigan, and Texas – as well as the mediocre NAEP scores seen in Louisiana’s charter sector – to argue that portfolio management and quarterbacks aren’t working.

I found their analysis to be overly narrow. Instead of taking some new evidence in and synthesizing this with the broad set of evidence available, they anchored on to one set of data points and made too strong of claims (especially in the titles of their posts).

Don’t Look at NAEP in Isolation 

Matt is right to point out that some states with fairly loose charter regulations saw a lot of charter gains in NAEP between 2009 and 2017.

I think this should modestly increase our belief that being loose on charter openings and closings can lead, over time, to a healthy charter sector.

But the story is not that clean.

This CREDO paper, which looked at charter school performance in Texas between 2011 and 2015, found a small positive effect in reading and no effect in math. Given that CREDO tracks individual students across time, and NAEP does not, the CREDO data should make us cautious in interpreting the NAEP gains as a huge victory.

Screen Shot 2018-04-24 at 9.57.03 PM

Another study found that it took Texas charters ten years to achieve the performance gains of their traditional school peers.

While it’s great to see the sector improving in Texas, perhaps with better regulation the sector wouldn’t have had to improve for ten years just to achieve neutral effects.

The story of the Texas charter sector is much more complicated than Matt’s piece indicated. The same is true for Arizona and Florida, where quasi-experimental research has found muted test score effects.

Given the different results of NAEP and CREDO, we should be trying to figure out the puzzle rather than claiming victory, as Matt tends to do.

Don’t Look at States in Isolation 

Looking at state level gains is not a great measure of whether portfolio and quarterbacks are working, as portfolio (to some extent) and quarterbacks (almost always) are city based endeavors.

For city based data, this CREDO study measured student learning effects in a bunch of cities across the country.

Washington D.C., Denver, New Orleans, and Newark all did very well. These cities are also some of the most mature portfolio cities.

Phoenix, Austin, El Paso, Forth Worth, Mesa, and San Antonio did not do well. These cities are all found in loose regulation states.

It hardly seems like a slam dunk to me that portfolio / quarterbacks are bad and loose regulation is good.

Of course, the CREDO analysis is not perfect: test scores aren’t everything and the virtual twin methodology may miss unobserved differences between students.

But looking at state based NAEP scores to make broad judgements on portfolio and quarterbacks is unwise, especially with so much other evidence available.

The portfolio / quarterback model seems to be doing some good in many cities.

Quarterbacks are a Step in the Right Direction  

Jay and Matt often criticize quarterbacks as vehicles for people who think they are smarter than everyone else (especially educators and families). I find this to be an overly simple critique.

Quarterback originated as a way to use expertise to aggregate and allocate philanthropy.

In many cities, philanthropists were funding low impact activities, often wasting it on the  pet projects of district leadership. A lot of money was spent for very little academic gain.

Quarterbacks have helped improve philanthropy: instead of just passively giving money to the district, philanthropists partner with expert management teams to try and launch and grow great non-profits.

I think this is a major improvement on the status quo. Of course, there are some drawbacks, and too much centralization of philanthropic capital poses risks. This is why I don’t think all of a city’s philanthropic capital should flow through one organization.

But quarterbacks are increasing, not decreasing, educator entrepreneurship and family choice. Yes, they do often use test score results selecting who to fund, but I suspect this will change if a better way to invest is developed over time.

In Sum

The NAEP data should not be ignored. It’s made me more open to the idea that looser regulations can lead to charter scale and quality, especially at the state level. And I found Matt’s data analysis to be quite helpful. I love it when smart people who think differently than me play with complex data sets and come to novel conclusions.

But I think there’s plenty of other state based evidence that should make us cautious, such as the CREDO Texas study.

I also think there’s a lot of evidence that the charter sectors in portfolio / quarterback cities are making a lot of gains. The NAEP data Matt and Jay site, which is state based and does not track individual students, is not convincing enough to make me deeply question our city based work.

All that being said, I look forward to reading more from Jay and Matt in the future.

I don’t take smart, critical friends for granted.

The best school district in the United States?

One of the joys of my job is the number of amazing emails that arrive in my inbox.

Below is an email (pasted with permission) from Scott Pearson, the head of the Washington DC Public Charter School Board.

On this blog, as well as on twitter, we debate a lot about regulation. We have a lot to figure out and these debates help me get smarter.

But leaders on the ground have to lead, always with imperfect information and complicated local contexts.

The DC Public Charter School Board has chosen to regulate the charter community fairly tightly on performance, but more loosely on other inputs. As Scott notes in his letter, over 40 charters have closed in Washington DC over the past decade. While I don’t know if this is right for every community, the DC charter community is providing a lot of great options for tens of thousands of children, and they have undoubtedly made DC a better city.

The continuity of the DC charter community’s success also reinforces my belief in the importance of non-profit governance. It’s hard to think of a better school district in the country, and I’m highly confident that a primary key to their success is their structure: the DC Public Charter School Board regulates and non-profits operate.

It’s a winning formula for kids.

___________________

Dear Colleagues,

We have faced nearly a year’s worth of bad news about DC Public Schools, from high teacher turnover, to faked suspension data, from inflated graduation rates, to the resignation of the DCPS chancellor, to residency fraud.  This steady drumbeat has undermined confidence in our traditional public schools – far more than is warranted in my opinion.  DCPS is a vastly better school system than it was in 2007 when Michelle Rhee and Kaya Henderson took the reins.

But I’m writing today about the other half of public schools in DC, the 120 public charter schools serving 43,340 students – nearly half (47.5%) of DC’s public school students.  Our story is mostly one of continued success, growth, popularity, and quality improvement.  I feel the need to write this because I fear that the bad news about DCPS is drowning out what continues to be a remarkable story of charter school success in our nation’s capital.

First, it’s important to say that not one of these bad news stories has been about public charter schools.  Our graduation rates check out – because for years the DC Public Charter School Board has audited the transcript of every graduating senior.  Our discipline data has never been questioned.  And there have been no allegations that anybody has jumped the lottery queue at a DC charter school.

Of course, we’re not perfect. In a sector as diverse as ours there will be unflattering news to uncover.  But at least so far none of the year’s bad news has touched DC’s charter sector.

Second, our quality keeps improving.   The NAEP “flatline” story just doesn’t apply to DC’s charter schools, as David Osborne and Emily Langhornewrite in The 74.  We were up this year in three of four grade/subject areas, with fourth-grade reading scores climbing five scale score points.  That’s more than any state in the nation.  And over ten years our growth overall is faster than any other state or district.  Charter scale scores have grown 17 points in 4th-grade math, 19 points in 4th-grade reading, and 12 points in 8th-grade math.  Each of these represents over a year’s worth of learning gains.   Only in 8th-grade reading are our scores disappointing, down 2 points over the past ten years and lagging far behind our big-city peers.

Our school leaders deserve much of the credit for this growth, but the authorizer gets some credit, too.  Since 2007 we have overseen the closure of 40 low-performing charter schools, all the while aggressively supporting growth for our highest-performing schools.

Third, our improvements are not driven by a change in our demographics.  Charter schools continue to enroll higher percentages of black and low-income students than does DC Public Schools.  And charters schools now educate the same percentage of students with disabilities as does DCPS – and higher percentages of our most disabled children.

Fourth, even as our quality improves, our schools have made remarkable progress reducing out of school suspensions and expulsions.  Out of school suspension rates are down by half since 2011-12, to under 7%.  And expulsions are down over 80% to less than 0.25% – about the national average.  We’ve achieved this through transparency and communications, not through mandates.  (Though, disappointingly, our city council is now threatening to regulate school discipline.)

Finally, demand keeps growing.  Despite the charter board adding nearly 9,000 charter school seats since 2013-14, the number of unique families on charter school waitlists has risen from 7,205 in April, 2014, to 11,317 in April, 2018.  Two-thirds of our charter schools saw their waitlist length increase from last year to this year.   Waitlists are in one sense a measure of our success because it shows families want our schools.  But is also a measure of our failure – and that of the District government – to provide our residents with enough quality schools and the facilities to house them.  We need to do better.

Thank you for reading, and please reach out if there is more information or context I can provide.

Scott Pearson

Executive Director

DC Public Charter School Board

What is California telling us about what parents want from public education?

I sit on the board of the California Charter School Association (CCSA), which is one of most effective charter associations in the nation.

Their data team put this slide together.

It’s a little complicated, but it’s very informative.

Screen Shot 2018-04-05 at 8.08.33 PM

To build the chart, CCSA looked at schools that are serving the same type of demographics and then compares their test scores.

A school that outperforms similar schools, gets a higher score (8-10 is really good); a school that underperforms similar schools, gets a lower school (1-3 is pretty bad).

This methodology – absolute test scores controlled for demographics – is imperfect, but it’s the best methodology you can use given California does not publicly report student growth scores.

Charter schools serving students in poverty outperform district schools on state tests 

43% of charter schools with higher concentrations of poverty outperform similar schools.

In California, if you are a low-income Hispanic or African-American child, you are more likely get a better education (as measured by test scores and parent demand) if you attend a charter school.

This is good news for the educators and families who are working together to create better educational outcomes in at-risk communities.

If parents are demanding schools with good test score impacts, the government’s response should be obvious: let more of these schools open.

Charter schools serving middle class students underperform district schools on state tests 

But not all charter schools are outperforming their peers on tests scores.

47% of charter schools serving middle class students perform worse than similar schools. And only 26% perform better.

So perhaps California should close some of these charter schools that serve middle class families? Research covered in this post shows that schools with negative test impacts tend not to have large positive life outcome impacts.

But here’s the odd thing: presumably, middle class families have a decent amount of information at hand when making school choices. It takes all of two minutes to scan Zillow or Great Schools to get a quick read on the absolute test score performance of any school in the state.

So why are all these relatively well resourced families sending their children to lower-performing charter schools as measured by state tests?

I’m not sure. It would be interesting to focus group and poll them to learn more.

And with regards to closure, while I surely disagree with middle class Californians on many policy issues, I’m not sure that I think I know enough about their children to close schools that have modest negative test impacts but high enrollment demand.

Charter schools don’t exist unless parents want their children to attend them 

One of the best features of charter schools is that they don’t exist unless parents choose them. No one is assigned to a charter school.

So what are we to make of this data where families in living in poverty are choosing schools with positive test scores impacts and middle class families are choosing schools with negative test score impacts?

I think the starting point should be to assume that families, on average, are in a better position to make an informed choice than government is.

Remember, government’s default assignment algorithm is to look at your family’s address and then assign your child to the nearest school. It’s not very nuanced!

My guess is that parent choice will outperform geographic assignment when it comes to finding great fits between kids and schools.

But I do think we should be open to the idea that parents, sometimes en masse, can make mistakes. And, at times, this can warrant government intervention.

Sometimes performance might be an indicator, such as when families keep sending their children to high schools with below 40% graduation rates and /or schools with extremely negative value-add scores. If less than half the kids are graduating, and those that do are barely literate, government should step in.

Sometimes lack of alignment with our nation’s professed values might be an indicator: certain public schools have at times been captured by groups, sometimes religious, that do not teach basic democratic values.

In these cases of significant performance or culture malfeasance, government should consider intervention, ideally by handing over management of the school to a non-profit organization that can achieve better results.

I don’t know enough about individual school performance to know if what’s happening in California with middle class families equates to education malfeasance, but I’m a bit skeptical.

My hunch is that once absolute test score levels surpass a certain floor (as they tend to in middle class schools), families just care a lot about other factors.

Even if I might make a different choice, I don’t know that the situation warrants government intervention.

When government should not intervene 

While it’s difficult to decide when government should intervene, it’s still pretty clear to me when government should not intervene.

When schools have both high demand and high test score impacts with students living in poverty, government should not prevent these schools from serving more students!

California should follow this common sense policy.

Unfortunately, too many school districts do not.

SaveSave

SaveSave

What do test scores tell us about schools?

Collin Hitt, Michael Q. McShane, and Patrick J. Wolf just published a report on the connection (or lack of connection) between test scores and long-term outcomes.

They looked at a bunch of school choices studies and tried to see if a school’s impact on student test scores was connected to its impact on student life outcomes.

Their conclusion: “at least for school choice programs, there is a weak relationship between impacts on test scores and later-life outcomes.”

Much of our K12 education policy is predicated on the idea that test scores are an important measure of school performance. If this is not true, behavior should change.

The Laura and John Arnold Foundation is currently funding more research on this question, and I’m eager to see what we find, especially with regards to income gains over time. This study was predicated on high school and college attainment being an indicator of long-term outcomes, but schooling isn’t always learning. So we should be careful judging school performance based on later school attainment, rather than income (or other measures).

But for now, here’s what’s on my mind after reading the study.

What do bad test score results tell us?

If I’m reading their report correctly (and I hope the authors correct me if I’m not), it seems rare that schools have a negative impact on test scores but a positive impact on long-term outcomes.

In the 126 study comparisons where test scores impacts were compared to high school or college outcomes, there were only 2 instances where a study found a significant negative impact on test scores and a significant positive impact on life outcomes.

It seems rare for a school to do really poorly on test scores but really great on life outcomes. I think the authors underemphasize this point in their paper.

What do mediocre test scores tell us?

It is much more common for schools to have a neutral (insignificant) impact on test scores but a significant positive impact on long-term outcomes. It looks like this occurred 32 times in the research review.

There may be a bunch of schools that don’t really impact test scores but are doing something that helps with long-term outcomes.

What do great test scores tell us?

There are no cases where a study found significantly positive test scores and significantly negative life outcomes.

So it seems rare for schools to jack up test scores but ruin kids lives. That’s good.

However, there were 17 instances of studies finding positive impact on test scores but neutral impacts on long-term outcomes.

So there seem to be some schools that are achieving good test results without translating these into great long-term outcomes.

How should this research affect regulation and philanthropy?

If these results hold, I think I will maintain my belief that we should replace schools with persistent very negative test scores. There appears to be little risk that these schools are really amazing schools. The negative test scores are a useful signal.

Yes, there might be other schools that are just as bad at life outcomes that are not closed because they achieve better test scores, but so long as we are closing schools that are not delivering great life outcomes, and opening schools that have a better chance of achieving great life outcomes, this seems like a worthwhile tradeoff.

But when it comes to expanding schools, if this research holds, I will rely less on positive test scores, and I think authorizers should do the same.

From an authorizer perspective, so long as a school does not have significantly negative test scores, perhaps the school should be able to expand so long as there is parent demand.

Philanthropy may also need to adjust by investing more heavily in school operators that show a positive impact on life outcomes (irregardless of test scores), and being willing to fund mediocre test score schools who either have high parent demand or who are using practices that are correlated with positive long-term outcomes (more research needed to determine what these might be).

I am very open to moving in this direction if research warrants it. The idea that it’s easier to tell a bad school than it is to identify a great school already matches my intuition, and deferring to parent judgment makes a lot of sense if we are not confident in our analysis of performance.