Category Archives: Testing

With the New ESSA, We’re Still Plugged into the Matrix

Leave a reply

4c73266777ce9631213fa52dd8a0b855e4f9e751

A great education leader who lives in Houston once said to me: “as long you’re worrying about state test scores, you’re still plugged into the matrix.”

His point: so long as public schools are held accountable via government tests, the incentives for educators will be about doing well on those tests.

If you believe performance on these tests is a useful measure of learning, then staying plugged into the matrix might be a good thing.

If you feel that parents, schools, universities, and employers are best suited to develop measures of learning, then you probably want to get out of the matrix and align incentives around different outcome measures.

In the long run, I think it’s probably a good idea to leave the matrix, so long as leaving the matrix is accompanied with a shift towards relinquishment, whereby educators can run schools and families can choose from these schools.

However, as long as we’re going to stay in the matrix, I think the two most important things are ensuring that the matrix is:

1) heavily weighted towards academic growth (rather than absolute scores) and;

2) that it identifies and acts on bottom performing schools and subgroups (where research indicates accountability helps the most).

Given that much discretion will be left to the states, time will tell if this matrix is a better than the previous matrix.

But either way, have no doubt about it: we’re still plugged into the matrix.

NAEP and the Great Convergence

4 Replies

Out of all the NAEP commentary, here’s what I thought what was missing: once you control for demographics, nearly every state performs about the same.

See below from Matt Chingos:

Screen Shot 2015-10-29 at 8.28.55 AM

A quick looks indicates that only 14 states perform outside of the +/- 2 month differential.

8 above: Massachusetts, Texas, Indiana, New Jersey, New Hampshire, Florida, Virginia, North Carolina.

6 below: Hawaii, West Virginia, Alabama, California, Michigan, Mississippi.

And only 4-5 states are outside of the +/- six month band.

A few thoughts:

  1. I have no idea if there are any commonalities between the 8 above the line or 6 below the line states that could lead us to any testable hypothesis about over and under performance.
  2. I haven’t dug into longitudinal data, but this feels like a great convergence of some sorts, with most states performing about the same once you control for demographics. I’d love for Matt Chingos to do his demographically adjusted rankings over time to see if this is in fact the case.
  3. All that being said, while most states are in the middle, the gap between the top states at the bottom states is pretty significant.

For the most part, this isn’t shocking to me, as my guess is that the general operational mediocrity of government operated systems mutes many policy differences that might exist across states.

But I’m not sure. Worth mulling over some more.

How I Responded on Email Chain About Rewriting NCLB

2 Replies
I was recently on an email chain where very smart people were debating the NCLB rewrite.
The debate had to do with whether or not the retreat from federally mandated accountability was a  good thing.
See below for my exact response:
_________
Some thoughts:
  1. The evidence on NCLB (annual testing, data transparency, etc.) is ok but not amazing. I think the upper bound I’ve seen is .2 effects over 6 year period.
  2. We don’t yet have rigorous data on teacher evals.
  3. We do have rigorous evidence on urban charter: ~.1 effects over a 3 year period (with the sector rapidly getting better each year – effects doubled over a couple year period).
  4. And now we have rigorous evidence on NOLA charter district reforms: ~.4 effects over a 5 year period; of course under unique circumstances.
All this leads me to believe (not with absolute confidence!):
  1. The federal charter program may end up being the most important federal education intervention. Tripling it from $250M to $750M will probably do more good for low-income kids than nearly every other federal program.
  2. The testing, accountability, eval movement will likely deliver real and modest gains. But it will never change the game. I am highly skeptical that mediocre school systems get excellent due to these backend levers.
  3. The 20-50 year game, I think, is about transitioning our public operated system to a publicly regulated but non-profit operated system + better teacher pipelines + tech.
  4. This is the .5-1 standard deviation game. It’s 75% supply and at most 25% accountability.
Just some thoughts. Obviously incredibly complicated. If there was a clear answer this many smart people wouldn’t be arguing about it.
-N
_________
You can hear clear undertones of the Allure of Order and the New Orleans theory of change.
I view it is as a near impossibility that accountability will ever deliver transformational results.
I wish our national policy conversation was 100x more about supply and 10x less about standards and accountability.

I Heart Ed Trust But Disagree with their Marc Tucker Take Down

Leave a reply

Kati Haycock just published a scathing post criticizing Marc Tucker’s argument that annual testing does not increase student achievement for poor and minority students.

Kati writes:

Screen Shot 2015-06-04 at 6.35.16 PM

Here’s what Marc had previously written:

Screen Shot 2015-06-04 at 6.36.43 PM

Some Refections

1. There are reasonable arguments for and against annual standardized testing.

2. I think Education Trust, and Kati in particular, have done very important work in shining a spotlight on school systems that fail poor and minority students. Education Trust has been able to do this, in part, because of data from annual testing.

3. I disagree with major portions of Marc’s argument, but I do agree with some of what he says.

4. In considering the issue, I do think we should give significant weight to the fact that many civil rights organizations are in favor of annual testing.

5. There are many of minority families that are against annual standardized testing. I personally know a few.

6. That fact that Marc Tucker is a white male should not prevent him from making (or prevent us from listening) to his views on the issue, especially given his expertise.

In Conclusion

So many times (especially on !), I’m temped to not intellectually grapple with the ideas of those with whom I disagree.

Failing to grapple with the ideas of those with whom I disagree has never served me well.

Tensions in Testing

1 Reply

For me, testing is a difficult issue.

On one hand, I view testing as a useful tool for understanding how well public schools are serving students.

On the other hand, I worry that centralized testing regimes inhibit innovation.

I wish this tension, as well as others, were broadly discussed in pieces that debated the issue.

Some other thoughts on testing that I wish received more attention:

1. Testing Can Increase Learning

Too often, testing is viewed as something that gets in the way of learning. However, research has shown that testing increases learning. The act of information retrieval cements knowledge. It is wrong to say that testing is “wasted” instructional time. Of course, content must be delivered as well as assessed, but assessment is part of sound instruction.

2. More Tests Can Fight Against the Narrowing of Curriculum

If you assume more instructional time will be spent on tested subjects, then increasing the number of tested subjects is a way to fighting against content narrowing. More testing can lead to more content coverage.

3. Testing is in Part a Substitute for Market Accountability

In most sectors of the economy, government does not hold companies accountable based on the quality of their products (unless they are breaking the law); rather, we assume consumers will be the best arbiters of quality. However, because our school systems are operated by (mostly) government monopolies, we can’t rely on market mechanisms to judge quality. One can make a reasonable defense for (1) public schools + testing or (2) voucher systems + no testing. But arguing for public schools + no testing risks being in a world with no real measure of quality.

4. Teacher Evaluations Require Annual Testing; School Accountability Does Not**

For value-added teacher eval models to be employed, you probably need annual testing. However, so long as every school has at least two tested grades, you likely do not need  annual testing to measure school performance. Getting rid of annual testing need not mean getting rid of school accountability.

My Tentative Ideal Education Governance and Testing Regime

Here’s what I think I’d want it to took like:

1. All schools are operated by non-profits that are on four year renewable contracts.

2. The government creates rigorous tests for every year of schooling*, but schools need not test every year. Rather every school must offer two tests across the grade spans it serves, so growth measure can be recorded. Additionally, every school must test its students at least once every four years.

3. To receive a contract renewal, a school must meet either a growth goal or a fairly high absolute goal.

4. To give easily consumable information to families, the government would assign letter grades to schools based on academic performance.

I think this could regime could increase innovation while still allowing for some oversight and accountability of public spending. However, this regime is predicated upon the non-profit operation of schools. I’d be hesitant to give up annual testing without significant increases in choice mechanisms.

*I’m open to the idea of government putting out an approved set of tests rather than forcing every school to take the same test, but I worry a lot about losing the ability to reliably compare schools, especially with regards to high stakes accountability systems.

** addendum: good back and forth on on whether or not you could do school growth models if you tested every other year or so. My guess was that you could, but if you can’t, I’d be more hesitant to give up annual testing.

The Connection Between Choice and Humility: Diane Ravitch Edition

1 Reply

china

Diane Ravtich had a long piece in NYRB this weekend. You can read it here.

Ravitch’s Argument

1. The United States has never scored well in international tests. And yet we lead the world in economic and military strength.

2. In 1983 a Nation at Risk came out with alarmist language that, in part, led to the national testing movement.

3. Since then, basically every president has said that are schools aren’t good enough, and that we need more / better tests.

4. In pushing this agenda, political leaders often point to the performance of Asian nations and cities.

5. Despite this rhetoric and accompanying strategies, our schools aren’t improving.

6. We should stop trying to emulate Asian nations: they over rely on rote learning instead of nurturing creativity.

7. Instead, we should shape our education policy based on two trends, increased globalization and technology, which will require giving more autonomy to both educators and students – as well as freeing them from the narrowing pressures of testing.

Where I Think Ravitch is Right

Little Correlation Between American Testing Performance and World Leadership: Ravitch is right to note that America has been a world leader despite performing at mediocre levels on international tests. Of course, world history is not a controlled experiment, so perhaps we’d have performed even better with high educational attainment, but, still, her point stands.

The Perils of Rote Learning: I agree that we should not fetishize the test scores delivered by rote learning and high-pressure cultures. At the very least, life is short and being a child in South Korea sounds miserable (surveys confirm they are amongst the unhappiest children in the world).

The Main Error in Ravitch’s Argument

Ravitch Provides No Evidence that Testing in Our Country has Reduced Innovation and Creativity: Ravitch’s main argument against testing is that it reduces innovation and creativity. Yet, she provides no evidence that this is occurring in the United States. I’m open to the possibility testing has had such an effect, but Ravitch provides no evidence that this in fact true.

Correlation Does Not Equal Causation: Instead of demonstrating how testing has reduced creativity and innovation in our own nation, Ravitch argues that China, which utilizes rote learning and testing, is not a very innovative and creative country. This may or may not be true (she provides little evidence), but, even if it is true, it’s not clear that testing is causing this lack of innovation. I think it’s more plausible that China’s rote learning and testing regimes are manifestations of their culture. It is unclear that changing the testing regime would change this underlying culture; just as it’s unclear that our own testing regime has in anyway affected our culture.

The Irony of Ravitch’s Proposals

Ultimately, the irony of Ravitch’s proposals is a familiar one: an education thought leader argues against top-down reforms and then offers up a vision of schooling which they think will be good for everyone.

Ravitch, in affirming the vision of her colleague, Yong Zhao, calls for:

schools where students produce books, videos, and art, where they are encouraged to explore and experiment … the individual strengths of every student are developed, not under pressure, but by their intrinsic motivation … schools where the highest value is creativity, where students are encouraged to be … confident, curious, and creative.

Aside from the fact that this is a pretty generic vision – and a vision that it is compatible with many standardized testing regimes – it’s unclear to me that every parent wants to send their child to a school where the highest value is creativity.

This seems to be what Ravitch wants.

But why should her vision of schooling be forced on everyone?

As it happens, there’s much I like about this vision. And, in arguing for this vision, I think Ravitch rightly identifies many of the core values of our nation (innovation, creativity, originality, and invention).

But let me add one more: liberty.

Instead of educators and families being forced to work in and attend schools that align with Ravitch’s vision, educators should be able to develop myriads of different types of schools that meet the different needs of the millions of children in our country.

Will some of these schools have creativity has their highest value? Probably so. Other may not.

Here’s the thing: I’d give up annual testing in a heartbeat if it meant our nation would transition to a system where educators could create non-profits to operate schools, parents could choose amongst these schools, and the government regulated the system with a less intense testing regime.

Unfortunately, this is not the vision Ravitch puts forth. Instead, she asks that we reject testing and instead adopt her own vision of public schooling.

In doing so, she ignores the connection between choice and humility.