Category Archives: School Performance

Stanford researchers find that New Orleans and Chicago are doing amazing things for black children

The New York Times / Upshot just took a massive Stanford researcher database and turned it into an easy to use webpage.

This was a big undertaking and these types of projects are some of the best of modern journalism. Kudos to them.

The research methodology is great in its scope (every city in America) though rather crude in its precision (raw proficiency gains from 2009 to 2015). It’s not as accurate as experimental and quasi-experimental research, but it’s still useful in taking a broad look across the country.

Which of the largest 200 school districts in the country saw the most growth?

Here’s the top 10:

Screen Shot 2017-12-06 at 7.00.57 AM

In 5 years, all of these districts achieved at least 5.5 years of academic growth. Of the 200 largest school districts in the nation, these are the highest performers.

Are these cities very similar to each other? 

Not at all. And this is where I think the reporters could have done much more. The New York Times journalists rightfully pointed out Chicago as an outlier, but they did little else to tease out the vast differences between these high growth districts.

Let’s take a look at these top growth districts by free and reduced lunch rates:

FRL3

See any differences?

Only four districts – Salem Keizer, Garden Grove, Chicago, and Orleans – have students bodies where the majority of students are economic disadvantaged.

And Chicago and Orleans are in a category of their own, with 80%+ students receiving free and reduced lunch during the years of the study. The task these districts face is 100x more difficult than that of the low poverty school districts.

Here’s the district African-American student enrollment percentages:

AA rate

See any differences?

None of the highest growth districts in the country except for Chicago and Orleans serve many African-American students.

New Orleans and Chicago serve many students whose lives are still shaped by the deep, generational poverty that is rooted in our country’s horrific history of slavery.

The fact that their academic growth is amongst the very best in the nation should be a huge cause for celebration and hope.

What we talk about when we talk about high growth districts

It took me about 45 min to create the above charts. I did quick google searches so the numbers probably aren’t perfect, but they paint a pretty clear picture.

Yet in the 10+ articles I’ve read about the Stanford research data set not one has mentioned that New Orleans is the only majority black school district to be in the top ten growth school districts in the nation. 

And in case you’re wondering if the New Orleans data includes all of its schools, it does. I emailed the researchers and they confirmed that the data includes all district and charter schools in the city boundaries.

So let’s give a big shout out to the amazing educators, students, and families who achieved unprecedented gains in learning in New Orleans between 2009 and 2015.

The New York Times might not have noticed, but hopefully others eventually will.

Quick feedback for the New York Times

Screen Shot 2017-05-05 at 8.44.11 AM.png

The NYT just published an article on NYC’s school choice system.

The article is worth reading for its qualitative insights into what it’s like to navigate the system. I have deep empathy for families that struggle to find great schools for their children. They deserve much better.

But the framing of the piece is flawed, and I hope other journalists don’t repeat this mistake in future articles.

The authors argue that school choice has not delivered on its promise because many students still don’t have access to great schools.

But school choice does not increase the supply of great schools; rather, it is a mechanism to allow families to choose from schools that already exist.

School choice is about access and fairness. You can assign families to schools based on their address, or you can try to create more just systems. I strongly believe we should do the latter.

But increasing equity of access will likely not lead to dramatic jumps in quality.

It is only be creating new schools, scaling the best schools, and improving existing schools that quality increases. This is not the job of a citywide enrollment system.

Moreover, if you increase access but restrict supply you well get frustration. And this is exactly what has happened in New York City. The city’s enrollment system persists, but its efforts to increase supply have faltered.

When NYC leaders have focused on increasing supply – both through the small schools movement and growing the charter sector – rigorous research found that school quality increased. The results of these efforts are detailed below.

In sum:

School choice is all about equity in access.

School supply is about creating better options.

We should not confuse the two, and we should not expect school choice to increase school performance in and of itself. It must be coupled with a deep focus on school supply.

Small schools results:

Screen Shot 2017-05-05 at 8.58.15 AM

Charter results:

Screen Shot 2017-05-05 at 8.53.01 AM.png

 

 

 

An Alternative Interpretation of the Fryer / Dobbie Texas Charter School Study

Screen Shot 2016-08-18 at 10.26.46 AM

Roland Fryer and Will Dobbie just published an excellent study on the Texas charter school sector.

But it’s unclear to me that they captured a very important implication of their research.

I. Study Overview

The study found that charter schools in Texas, on average, have no impact on test scores and a slightly negative impact on earnings.

More interestingly, the study found that No Excuses charter schools increase test scores but only have a small and statistically insignificant impact on earnings.

Their paper ends with this cautionary statement:

Screen Shot 2016-08-18 at 10.34.38 AM

II. Walking Through Low Effect Size and High Effect Size Schools 

The famous Anna Karena quote goes something like this: “Happy families are all alike; every unhappy family is unhappy in its own way.”

I think the opposite is true of schools.

When I visit low effect size schools, I am often saddened by the level of dysfunction. Students walk the halls aimlessly, teachers seem woefully unprepared for working in a low-income environment, and the principal generally spends her day putting out fires.

When I visit high effect size schools, I’m often struck by how different they are. While most hit the basics of a calm culture and thoughtful instruction, they vary greatly in atmosphere, curriculum, and staffing models – as well as the overall student experience. A Summit school is very different than a Collegiate Academies school, despite both achieving high effects. Even No Excuses schools can feel fairly different from each other, though they do tend to gravitate around some core practices (that Fryer has helped illuminate).

I also think I would struggle mightily in a blind walk through of .1 and .2 effect size schools; it is highly unlikely I would be able to tell you which school has which effect.

So while it’s easy to identify schools that are a total mess, it’s a little difficult to tease out what’s going well in non-dysfuctional schools, as well as to distinguish between high-performing and very-high-performing schools.

III. Bad Schools Have Bad Effects on Earnings, Good Schools Have Neutral Effects on Earnings

I found this to be the most interesting chart in the study:

Screen Shot 2016-08-18 at 10.33.43 AM

What you see here is that going from (-.2) to (0) effect really matters for earnings. This is indicated by the rising slope in the bottom left quadrant.

Interestingly enough, once you hit (0) effect, going to (.2) effect has little effect on earnings. This is indicated by the relatively flat slope in the the bottom right quadrant.

In short, getting rid of bad schools could have a major effect on the earnings of graduates in an education system (assuming our economy is not a zero sum signaling game).

In a sense, this fits my experiences in spending time in schools. It’s very easy to see how a totally dysfunctional environment could negatively impact students, whereas it’s a little more difficult to tease out the additional impact on students once the basics are in place.

IV. Portfolio Management: What Happens When Charter Schools Grow?

In a world where states and districts are managing their portfolio of schools, the growth of functional schools will be accompanied by the phasing out of dysfunctional schools.

In the best possible world, the growth of new effective charter schools will be accompanied with a reduction in under-performing traditional and charter schools.

Overtime, a system can potentially rid itself of failing schools.

This is what happened in New Orleans.

Screen Shot 2016-08-18 at 12.37.37 PM

While the above analysis is more weighted toward absolute scores (rather than effect sizes), my hunch is that the story would stand with effect sizes as well (I have not run this data yet).

I think much of New Orleans’ gains were driven by the phasing out of failing schools.

It is much less clear to me that schools in New Orleans, to date, have figured out to crack the code of creating schools that are radically superior to your average functioning traditional school.

Hopefully they will.

V. No Excuses Charter Schools May Allow Us to Eliminate Failing Schools and Raise the Aggregate Earnings of Low-Income Students in the United States 

So another way to interpret this study is that the growth of No Excuses charter schools could be the key to eliminating failing schools and raising wages of low-income students who would have otherwise have attended failing schools.

Two things would have to hold true for this to be the case: (1) government action or family choice lead to the phasing out of failing schools and (2) No Excuses schools can maintain their neutral effects on earnings even if they enroll the most challenging students from the phased out failing schools.

In other words, for now, the importance of charter school growth might be much more directly tied to eliminating failing schools rather than vastly outperforming functional district schools.

If this is right, No Excuses charter schools might still very well be the most important education reform of the past quarter century.

Is No Excuses or Personalized Learning the Low Hanging Fruit of School Improvement?

On average, I visit a school every other week or so. For the most part, these schools are equal to or better performing than the median urban district school.

During these visits, one question I usually mull over is this: if I was leading the school, what would I focus on to drive the next phase of improvement?

Often times, what the school leader is focusing on and what I would focus on are at odds.

I don’t have extremely high confidence in my analysis, so consider the below speculative.

____

Here are the things I most often here school leaders saying they need to improve on: personalization, student ownership, and critical thinking. Tactically speaking, this often leads them to experiment with new models of instruction and technology.

All good things.

But I’m often thinking that the school really needs to get better at: instructional delivery, higher ratios of student intellectual engagement, and more effective use of small group instruction.

Most school leaders seem to believe that they have the basics down and need to go from good to great.

I tend to think that most schools are mediocre at the basics of things such as cold calling, wait time, efficient time on task, and tutoring – and the other hall marks of the no excuses model.

___

So here’s some questions on my mind:

  1. Should the median charter school be focusing on getting better at the basics of the no excuses model or experimenting with deeper innovation?
  2. If it’s true that the median charter school is still mediocre at the no excuses basics, what should we take from this? That high fidelity to the no excuses basics is operationally hard to scale for either intellectual, emotional, or human capital reasons? That many leaders don’t think the no excuses basics work?
  3. Is there a progression of improvement (i.e., you need to get the basics right before you work on deeper innovation) – or does shifting to more innovative models allow you to bypass the no excuses basics and still get academic gains?

___

My hunch is this: only the top tier charter organizations and the very best entrepreneurs should be working deeply on the margin of innovation.

Most charter schools should be working on the margin of better adoption of the tenets of the no excuses model.

Once new models are hammered out and refined – and get better results than the no excuses model – then the median charter school should begin adopting these new models.

But not before that.

In sum, I think better fidelity to the no excuses model is the low-hanging fruit of school improvement.

Maybe I’m wrong? Maybe I’m very wrong?

Are US Public Schools Better Than They Have Ever Been? If So, What Should We Do?

better

This claim was put forth by Vox.com in an article headline, which I assume means they support this thesis.

The arguments were provided in interview format by Jack Schneider.

Overall, I agree with Jack that schools in the USA are getting better.

Jack uses two data sources to make his argument: parent perception and NAEP scores.

I find the latter argument much more persuasive than the former.

Parent Perception

Jack notes that families rate 50% of schools in their neighborhoods as an “A” or a “B” – while they only rate 25% of schools across the nation at this same level. He takes this as a sign (I think) that families have an accurate read of their local community schools, but view the nation’s schools as worse off because of all the scare tactics put forth by education reformers.

This strikes me as a weak argument for numerous reasons.

First, it is not a historical argument. It tells us nothing about whether or not schools are better now than they were before.

Second, Jack provides no statistical evidence that parents can accurately rate the schools in their community. In fact, later in the piece, he notes that “people don’t really have good information school quality.”

Then why are using perception as a marker of national school quality?

One could easily create an argument based on cognitive dissonance (“I send my child to this school therefore it can’t be bad”) to make a case for why parents might be susceptible to biases, especially in the absence of easily understandable quality markers, such as letter grades.

Lastly, even if we grant that these measures are useful, are we happy with the idea that families only think 50% of the schools in their communities are a the “A” or “B” level? That would mean that half the kids the community go to subpar schools.

Test Score Data

Jack notes that test score data (I assume he is referring to NAEP) show positive trends.

This is positive news. And it’s why I think are schools are getting better. I also agree with Jack that are public schools are serving a harder to educate student body, which makes these gains even more noteworthy.

Here’s some data:

Screen Shot 2014-10-13 at 9.43.24 AM

Are We Good Enough? 

Even if we acknowledge that schools have improved, we might still worry that we’re not good enough.

International test data might be one way to measure this.

Jack argues that we shouldn’t be too worked about the fact that many countries outperform us on PISA:

I say, OK, tell me one question that’s on the PISA (the OECD’s standardized tests taken by students in more than 60 countries and economies). And that’s where we stop. People don’t know a whole lot about it, but it is a nice piece of evidence that confirms this thing they already believe because they’ve heard it so many times.

I’m not sure that the fact that someone can’t recite a PISA question is a sound reason to disregard a layperson’s concerns. I also seem to have more faith than Jack that test scores give a lot of meaningful information about student learning.

Overall, I do find PISA performance to be a useful indicator, though there’s not a ton of room here to go into why. I don’t think that a country’s PISA performance tells us all we need to know about the health of the nation (the USA has never scored particularly well, and our economy remains one of the strongest in the world). But, all things being equal, I think the our national outlook would be better if our students were performing better (and we maintained our other national advantages).

That being said, I think we can answer the question “are we good enough?” by looking at our own internal data. Our national college achievement rates are generally flat; are urban school systems produce very low absolute results; and we have real instances of pockets of positive results to demonstrate that this need not be the case.

Where Do We Go From Here?

Jack’s answer:

Let’s just figure out how to build capacity in individual schools. I would really love to hear some reformer say, my really big issue is for every school to figure out what it needs to improve and I want to create a system so that school can get the things it needs in order to improve. That’s the only thing that I think is scaleable, is talking about how to improve the capacity that schools have to improve themselves. Districts and states can absolutely play a part in that, but this idea that we’re going to stumble on some magic solution for schools that we haven’t found in the last 200 years is I think pretty shortsighted. Certainly it’s simplistic.

To be honest, I cannot summarize his argument, because I do not understand it. Taken literally, I suppose it would mean that every school would generate an improvement strategy, and then local and state governments would then support this strategy.

If that is Jack’s argument, I’m highly skeptical it will work.

My thoughts on how we might improve can be found here.

I admit that my solution is untested at scale.