A weak SIGnal: flawed research means we don’t know if SIG worked

I have a piece in Education Next about the study that came out the $7 billion federal school turnaround program.

Everyone is saying that the study proves SIG didn’t work.

I disagree.

My main argument is here:

In detailing these results, the authors note:

“The smallest impacts our benchmark approach could detect ranged from 0.19 to 0.22 standard deviations for test score outcomes, from 0.15 to 0.26 standard deviations for high school graduation, and from 0.27 to 0.39 standard deviations for college enrollment.”

Now, look back up at urban charter effects and you’ll see the three year results in math are about at the floor of what the SIG study could detect, and the results in reading are much lower than what the SIG study could detect (the SIG study also tracked children for 3 years).

So even if SIG achieved the same effects as urban charter schools the study may not have been able to detect these effects. 

It seems pretty unfair for charter (or voucher) champions to call SIG a failure when SIG might have very well achieved near the same results as urban charter schools.

My conclusion:

Until I see results that show that SIG worked, I won’t change my prior belief that SIG funds would have been better spent on high-quality charter growth.

Moreover, neither the existing research base nor theory warranted a $7 billion spend on district turnarounds, so even had the intervention worked I still would consider it a lucky outcome on an ill-advised bet.

But I also won’t claim that SIG failed.

Due to poor research design, we simply don’t know if that’s true.

The study authors, reporters, and commentators should walk back their strong claims on SIG’s failures.

At the same time, we should all keep advocating for government investment amounts to be in line with the existing evidence base.

If we have no reason to believe something will work, we should not spend $7 billion.

Too often, moonshots garner more status then they deserve.

Read the the whole piece here.

2 thoughts on “A weak SIGnal: flawed research means we don’t know if SIG worked

  1. Pamela J Kingsley

    $7 billion spent/wasted and we’re still looking for a pony in that pile?
    Pearson and other school carpetbaggers were the only winners. Overnight, they created their bevies of “expert” consultants to reap the monetary benefits of SIG.
    In Kansas City, desperate district leaders (under state pressure) paid Pearson more than $1,000,000 per year, per school for 3-5 years. And that was just the consultant fees — in addition to the district spending millions on Pearson’s full array of sure-fire materials. Results: Zip.
    Pearson’s excuse: the schools don’t implement our programs with fidelity.
    Only a public school consultant could say that with impunity.

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s