Category Archives: NAEP

NAEP and the Great Convergence

Out of all the NAEP commentary, here’s what I thought what was missing: once you control for demographics, nearly every state performs about the same.

See below from Matt Chingos:

Screen Shot 2015-10-29 at 8.28.55 AM

A quick looks indicates that only 14 states perform outside of the +/- 2 month differential.

8 above: Massachusetts, Texas, Indiana, New Jersey, New Hampshire, Florida, Virginia, North Carolina.

6 below: Hawaii, West Virginia, Alabama, California, Michigan, Mississippi.

And only 4-5 states are outside of the +/- six month band.

A few thoughts:

  1. I have no idea if there are any commonalities between the 8 above the line or 6 below the line states that could lead us to any testable hypothesis about over and under performance.
  2. I haven’t dug into longitudinal data, but this feels like a great convergence of some sorts, with most states performing about the same once you control for demographics. I’d love for Matt Chingos to do his demographically adjusted rankings over time to see if this is in fact the case.
  3. All that being said, while most states are in the middle, the gap between the top states at the bottom states is pretty significant.

For the most part, this isn’t shocking to me, as my guess is that the general operational mediocrity of government operated systems mutes many policy differences that might exist across states.

But I’m not sure. Worth mulling over some more.

Do Flat High School Scores = No Useful Learning Gains? I’m Not Sure.

Both Kevin Drum and Rick Hess recently wrote about flat / declining high school learning gains.

Kevin argues that gain in K-8 are being washed out by poor high school performance, while Rick points to inconsistencies in those who are caught in trap of arguing both K-8 is awful (it washes out pre-k gains!) and K-8 is great (it’s the high schools that mess everything up!).

Kevin writes:

In the end, though, it doesn’t matter what the score is in the sixth inning if your bullpen consistently blows big leads. What we care about is how well educated our kids are when they leave school and enter the world. Until our high schools are able to build on the big gains they’re inheriting from middle schools, we’re not going to see any improvement on that score.

Rick writes:

 In any event, I’m curious if those who are raising the “alarm” over high schools that are fumbling away our K-8 gains have now decided that Head Start’s initial results are what counts, and the actual problem is K-8 fumbling away our hard-earned pre-K gains.

But I think there might be a flaw in this line of thinking.

It seem plausible that learning how to read / do math at an 8th grade level is actually very useful in life (assuming this equates to a basic level of numeracy and literacy), regardless of whether or not one ever masters high school content.

The same is clearly not the same for pre-k learning, which doesn’t really give you a leg up in life if you don’t continue to learn.

In this sense, 8th grade achievement should be considered independently of high school failure; i.e., learning how to multiply is useful even if you never master Algebra II.

Following this logic, increasing 8th grade scores and flat high school scores could indicate real learning improvement that is not subject to washout.

In order to understand if this was true, we’d want to test high school students on 8th grade content to ensure the learning stuck, which I don’t know if we’re doing.

But the fact that high schoolers are not increasing their performance on high school content doesn’t seem to be evidence that they’ve lost their 8th grade knowledge.

Am I missing something?

Admittedly, I’m not an expert in NAEP methodology, so perhaps flat high school scores does indicate 8th grade wash out. I’d just to love see this case explicitly made.

Until then, I’ll sleep a little better knowing that 8th grade scores up.