Good news for Camden’s children

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter, innovation, and traditional schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

CREDO’s analysis measures how much a school helps a student grow over the course of a year. They do this by comparing students in the city to similar students across the state.

CREDO presents its findings in standard deviations. A useful way to understand these impacts is to translate them into extra days of learning, based on a 180 day school year.

Screen Shot 2019-07-08 at 7.45.53 PM.png

As the chart above shows, a .15 standard deviation impact equates to about an extra half year of learning.

The results just came in Camden.

Camden Reform History

The state intervened in Camden schools in 2013. You can read more about the effort in this good New York Times piece.

One of the major innovations in the takeover was the creation and expansion of Renaissance schools. Renaissance schools are governed by non-profit organizations but must serve all students in the neighborhood. They are sort of a hybrid between charter schools and traditional schools.

The Renaissance reform effort was also coupled with improvements to traditional schools. Schools became safer and academic improvements were implemented across the city.

The city also created an online unified enrollment system to help families find the best public schools for their children.

Large Citywide Improvements

Camden’s city level effects are large.

In just two years, scores are up ~.15 standard deviations in math and ~.05 standard deviations in reading (compared to similar schools across the state).

To put this in context, over five years, New Orleans achieved a .4 standard deviation effect. These city effects were the largest the researchers had seen. Camden may achieve similar results. The math results are on track to mirror the gains seen in New Orleans.

It’s pretty incredible to see students learning so much more so quickly. Effects this large are a good signal that students are getting smarter in literacy and numeracy.

Screen Shot 2019-07-08 at 7.48.31 PM.png

 

Renaissance and Traditional Schools are Improving the Fastest

Renaissance schools are the highest performing sector in Camden, outperforming similar schools across the state in both reading and math. They also improved by over +.1 standard deviations in both subjects over the last year of the study.

The Camden traditional sector, though lower-performing, has improved. District schools have seen large improvements in Math (+.2 standard deviations) and modest gains in reading (+.06 standard deviations).

The charter sector continues to outperform the district, though it has seen a decline in its learning gains relative to the state over the past few years

Screen Shot 2019-07-08 at 7.50.41 PM

Screen Shot 2019-07-08 at 7.50.51 PM

Will Learning Improvements Lead to Better Life Outcomes for Children?

In New Orleans, we began to worry that gains in test scores, while important, would not translate into better life outcomes for students. Unfortunately, There were not enough post-secondary programs in the city that could help high school graduates prepare for meaningful careers.

Many cities across the country also struggle with this issue.

Recently, the former superintendents of New Orleans and Camden announced they were launching a new organization, Propel, to help high school graduates transition to good careers.

This promising effort, if it works, will help students capitalize on their increased numeracy and literacy skills.

Mission Not Accomplished

The Camden reforms are barely past their fifth year. The city is still home to struggling schools. Absolute achievement remains low. And the district remains under state takeover.

Hopefully, over the next five years the city’s schools will return to local control and continue to improve. And all of the work will translate to better life outcomes for students.

All public schools (traditional and charter) rising in Newark

When more students enroll in non-profit charter schools, what happens to the students who remain enrolled in traditional schools?

This is one of the most contentious questions in public education right now.

Past research has shown that increased public charter school growth does not negatively affect the academic performance of traditional public schools.

But much of this research covers geographies that don’t have that many charter schools.

An open question is whether the effects of charter growth on traditional public schools will change as charters serve more and more kids in a district.

In Newark, nearly 40% of students attend charter schools.

At this scale, non-profit schools have given families a lot more choices to find a good fit for their children.

But they have also put real academic and financial pressure on the traditional system.

So what’s been the effect?

New Jersey ranks all of its school districts based on academic performance. The state runs a couple of different types of analysis: ranking all district statewide and then also ranking districts based on those that have similar levels of poverty.

See below for the results for Newark citywide, Newark traditional system, and the Newark charter sector.

Newark’s Overall City Rank is Rising 

Newark has shot up from the 39th percentile to the 78th percentile amongst the thirty-seven highest poverty districts in New Jersey,

In the 100 highest poverty districts, Newark has moved from the 18th percentile to the 50th percentile.

When comes to all districts, Newark performs poorly, though it has made major progress in the past five years. This progress comes after a fairly long period of stagnation.

Screen Shot 2019-06-17 at 9.12.00 AM

Newark Traditional Public Schools are Improving at a Healthy Rate

Newark’s traditional schools have made major improvements over the past five years, after being fairly flat in previous years.

When it comes to the highest poverty cities, the district’s traditional public schools have moved from the 20th percentile to at or above the 50th percentile.

They also have also seen gains the other performance rankings, though overall gains when compared to all New Jersey districts are fairly modest.

Screen Shot 2019-06-17 at 9.12.23 AM

Newark Charter Schools have Nearly Caught the State Average 

Newark charters are achieving at very very high levels.

Taken together, they are the top performing high-poverty district in the state.

Even more impressive, Newark’s charters have risen to nearly the 50th percentile in the entire state of New Jersey.

New Jersey is one of the wealthiest states in the nation.

Students in Newark charter schools, who are mostly children of color living in poverty,  are performing as well as their much more affluent and privileged peers.

Screen Shot 2019-06-17 at 9.12.10 AM

Still an Open National Question

Newark is just one of many cities that are now home to large non-profit public school sectors.

We are working with researchers to study city level and sector level effects across many of these cities.

In the fall, I hope to have a more comprehensive write-up on these results.

But, for now, it’s good to see all schools rising in Newark.

Hopefully these results will hold true across many more cities.

Bloom, New Orleans, and Effect Sizes

Six months ago Matthew Kraft published an excellent article on effect sizes.

I worked in education for five years before I had any understanding of research design and reporting. I wish Matt’s piece was around a decade ago.

His article is a bit dense if you’re trying to just wrap your head around the issue, so consider this post a lay person’s intro to Matt’s piece and the subject itself.

If you catch any mistakes, please do let me know. I’m still learning.

Why are effect sizes useful?

Consider currencies. Currencies are useful because they allow you to easily compare prices across various goods. Instead of having to constantly refer to one set of goods in relation to another set (ie, three apples are worth the same as four oranges which is worth the same as three paperclips), we can use the same unit (dollars) to compare a bunch of different goods.

Effect sizes serve the same function. They help us easily compare the magnitude of the impact of a bunch different interventions. We can do research on graduation rates, test scores, suspension rates, or whatever we want, and then we can convert our results into an effect size to help us compare how big of an impact we had.

Effect sizes are the unit of currency for measuring impact.

What is an effect size?

Many effect size calculations in education research are expressed in standard deviations.

A common formula to determine the effect size is:

(mean of experimental group – mean of control group) / standard deviation

Let’s say we trying to find the effect size of a new math curriculum on test scores. We might give half the population the new curriculum, half the old curriculum, and then see what the difference is.

Let’s say the difference is +5 pts out of a 100 for the students using the new curriculum. The curriculum “worked.”

But what does that mean?

We now want to know if +5 pts is a big deal. This is where the standard deviation comes in.

A low standard deviation means there is very little difference in the population (everyone is scoring about the same score). A large standard deviation means there is a wide spread in scores.

Because the standard deviation is the denominator in the formula, the smaller it is, the large the effect will be for any given difference between two groups.

In other words, if everyone is scoring between 62 and 65 out of a hundred, and you jump five points, you could go from the bottom 1% of test takers to the top 1% of test takers.

Because the standard deviation is low (small spread), a modest jump leads to a big effect.

What is a large effect?

This is where Matt’s paper is particularly useful.

Much of the previous literature on effect sizes made many mistakes:

  1. Sample sizes were ignored.
  2. Duration of treatment were ignored.
  3. Time elapsed until measurement was ignored.
  4. Cost was ignored.

Taken together, scalability of interventions was ignored. This had the unintended consequence of setting the bar too high for what should be considered a large effect size.

Bloom’s 2 standard deviation effect 

You may have heard of Bloom’s 2 sigma tutoring intervention. This result is taken to show that 1-1 tutoring can have a 2 standard deviation (very large!) effect.

But Bloom’s study design was the following: take dozens of 4th, 5th, and 8th graders; give them 1-1 tutoring in discrete subjects like cartography or probability; and then test them on what they learned after 3-4 weeks!

It’s much easier to squeeze out a big effect under these conditions.

These types of small sample studies led to a research norm where an effect size had to be .8 standard deviations for it to be considered large.

New Orleans’ .4 standard deviation effect 

Contrast Bloom’s study to Doug Harris’ study on the New Orleans education reforms.

The New Orleans study covered tens of thousands of students. Students received the treatment across all major subjects, including math, reading, science, and social studies. The treatment lasted multiple years. And students were tested once every year in each subject.

It’s a lot harder to make large gains under these conditions, especially when the intervention costs under 20% of 1-1 tutoring.

Doug’s study found .4 standard deviation effects for New Orleans students over a five year period.

In his paper he wrote that he was “not aware of any other districts that have made such large improvements in such a short time.”

To summarize:

  1. The standard bar for a large effect was .8 standard deviations. This was irregardless of sample size, length of treatment, measurement proximity, or cost. The bar was poorly constructed.
  2. New Orleans achieved a +.4 standard deviation effect on test scores.
  3. Researchers had never seen a citywide effect this large before.

There are two ways to interpret this.

  1. The previous .8 standard deviation bar was way too high for large samples.
  2. The New Orleans effect, despite being relatively large for district improvement, is still so absolutely small that we should not be too impressed.

Was the New Orleans effect too small?

The +.4 standard deviation effect equates to the average New Orleans student moving from the 22nd to 37th percentile in performance.

For any individual, this might or not be life changing. But in the aggregate this means the average New Orleans student roughly went from a borderline high school dropout (bottom 20% of performance) to a student who has a real chance to enter a two year or four year college (modestly below average performance).

Across a large population, this is a pretty big deal.

We should pay attention to a city level +.4 standard deviation increase in test score. If this effect (or even one somewhat lower) can be scaled, kids across the country will have a better chance at leading a good life.

Of course, academics and test scores are just one piece of the puzzle of economic mobility, but they are an important piece. Schools with negative effects on test scores tend not to deliver great long-term life outcomes for kids.

Matt Kraft’s proposed effect size scale

When it comes to large interventions, Matt argues we should get rid of the .8 standard deviation benchmark.

I agree.

Matt proposes the following rough scale:

Small effect: less than .05

Medium effect: .05 to .2

Large effect: .2 or larger

Matt reviews a bunch of educational studies to help come up with this table. While I don’t love that it averages a bunch of very different studies, at the very least it sets conservative estimates on effects and cost (given that averages include studies that don’t meet the highest bar for sample / duration / etc.).

Screen Shot 2019-05-29 at 5.10.56 PM.png

Take a look at where .4 standard deviations shows up. The New Orleans reforms are in the 90th percentile of magnitude but the 60th percentile of costs. New Orleans increased it’s pup-pupil by $1,400 in the years following Katrina, though it’s not clear to me that the money is what really drove the effect. But even if you assume it did, the results pass a ROI test.

Again, the New Orleans impacts are pretty remarkable.

In considering impact, cost, and scale, Matt also provides the following matrix:

Screen Shot 2019-05-29 at 9.49.25 PM.png

New Orleans does well.

In Sum: Toyotas > Ferraris 

When it comes to effect sizes, be very careful to review sample sizes, treatment duration, measurement proximity, and cost.

Holding out for .8 standard deviation effects is foolish. These effects will rarely occur and when they do they tend to be very hard to scale.

When it comes to large scale interventions across medium term time frames, effects above .2 standard deviation warrant our attention.

The most realistic path for broad academic gains is to look for meaningful jumps in student performance that are caused by an intervention that has a real chance of scaling over time. And then testing and scaling and testing and scaling.

In other words: Toyotas > Ferraris.

Baton Rouge’s charter school turnaround

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter and innovation schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

The results just came in for Baton Rouge.

The Historical Underperformance of Baton Rouge’s Charter Sector 

After Hurricane Katrina, New Orleans’ public charter schools drove impressive achievement gains for the city.

Baton Rouge civic leaders and educators tried to replicate this strategy. It didn’t work. The charter schools that opened performed poorly.

A typical response to these weak results would have been to abandon the strategy.

But this is not what happened in Baton Rouge.

Civic leaders, led in large part by the leadership of New Schools for Baton Rouge, came to the conclusion that the charter strategy was right but that the execution was flawed.

So they began the hard work of recruiting new charter leaders as well as working with struggling schools to wind down their operations.

7 Years Later

New Schools for Baton Rouge was founded in 2012. It took many years to raise money, build a civic coalition, and recruit great educators.

Now, in 2019, we’re beginning to see the results of the work.

The slides below are a bit hard to read, but they capture the achievement effects of magnet, charter, and traditional schools in Baton Rouge between 2014 and 2017.

The study compared students in these sectors to their peers across the state (controlled for demographics and other salient factors).

Math Effects

Screen Shot 2019-05-11 at 9.08.07 AM

Reading Effects

Screen Shot 2019-05-11 at 9.07.56 AM

If you look at the small dotted lines tracking the [C] results, you’ll see major improvements in the Baton Rouge charter sector.

In 2014, the charter sector had -.13 effects in math (yellow lines) and -.16 effects in reading (blue lines).

These are pretty bad results.

But only a few year later the charter sector had gone from negative effects to neutral effects.

Improving an entire sector by ~.15 standard deviations in just three years is an impressive feat. Keep in mind that New Orleans achieved ~.4 standard deviation impact in the six years after the storm; so to grow .15 standard deviations in just three years is nearly on par with the growth we saw across the city in New Orleans.

While test scores aren’t everything, it’s rare that schools with very negative test effects deliver positive long-term life outcomes.

So this turnaround of the Baton Rouge charter sector is wonderful to see.

Lessons for Other Cities

In CREDO’s study of urban charter sectors across the country, around 20% of cities had negative effects for their charter sectors.

If I were a leader of one of these cities, I’d get on a plane and take a trip to Baton Rouge.

Baton Rouge leaders are showing that it’s possible to turnaround a city’s charter sector. The strategy is fairly simple. Open up annual cohorts of good schools and replace underperforming schools.

But the execution is difficult.

It took a great board and team at New Schools for Baton Rouge to give good charter operators the confidence they could grow in a city that had a struggling sector.

It took major investments in teachers and leaders to grow the talent base to support operator growth.

And it took a coalition of advocacy organizations and political leaders to create the policy conditions to rebuild the sector with a much stronger emphasis on performance.

Even cities with good charter sectors could learn much from Baton Rouge’s wholistic approach.

Of course, the goal for Baton Rouge charters is not to achieve neutral effects.

But given the baseline, neutral is the first stop toward positive.

And given the rich entrepreneurial ecosystem that’s been built in the city, I’m optimistic that educational opportunity will continue to expand across the city.

Not just a great teacher. Not only a few good schools. An entire city.

Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter and innovation schools in select cities across the country.

Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.

The results just came in for New Orleans.

Worries of Stagnation in New Orleans

Many observers, including myself, have been worried about the stagnating proficiency scores in New Orleans.

Screen Shot 2019-05-08 at 12.45.06 PM

One of the major problems of looking at proficiency cut-offs is they are a somewhat arbitrary cut-off line. For example, if a student started off extremely below proficient, but then improved to one test score question away from being proficient, the student would still show up as not proficient.

This is why Arnold Ventures funded the CREDO analysis. CREDO’s methodology allows us to examine achievement growth across all of the performance spectrum.

CREDO Analysis of New Orleans 

Fortunately for kids in New Orleans, CREDO found that New Orleans schools are continuing to outperform the state on academic growth.

Here is the most important slide from their analysis:

Screen Shot 2019-05-08 at 12.49.21 PM.png

The statistically positive achievements effects indicate that students in New Orleans are catching-up with the state somewhere in the range of .05 to .1 standard deviations a year.

To put these gains in some context, the black white achievement gap in the United States is .8 standard deviations.

These are strong, citywide annual gains. And it’s very important to emphasize that these are city level effects.

In most cities, the longer poor children stay in the system the further behind they get. In New Orleans, the opposite is true: the longer you enroll in New Orleans public schools, the closer you get to your peers across the state. 

This is not the case of a few charter schools doing well. This is about the whole city making gains.

The goal of city based education reform should be for all kids to benefit, not just a few.

And that’s happening in New Orleans.

Hopefully, these gains will continue to compound and, eventually, further raise citywide proficiency, graduation, and post-secondary success rates.

Is it ok that Lebron’s school chooses not to serve the most at-risk children in Akron?

I don’t have strong emotional attachments to athletes, but as far as athletes go, I like Lebron James.

Lebron got thrown into the spotlight as such young age, has accomplished amazing athletic feats, and is committed to trying to make his hometown better.

The New York Times recently wrote a piece on Lebron’s new public school in Akron.

Much could be written on how and why the school was covered, but I’ll stick to one point that I felt has not been covered: Lebron’s school refuses to serve students in the bottom 10% of performance.

From the article:

I Promise students were among those identified by the district as performing in the 10th to 25th percentile on their second-grade assessments. They were then admitted through a lottery.

Clearly, serving students in the 10th-25th percentile is not easy.

But serving kids in the 0-10th percentile is much more difficult.

A common critique of charter schools is that they don’t reach the very hardest to serve students.

There has not been a similar outcry against Lebron’s school.

Can you imagine the NYT headline if top charters had a formal policy about not serving students in the bottom 10% of performance?

___

On the substance of the issue, I’m sympathetic to Lebron’s approach. Serving students in the 10th-25th percentile well is both very hard and very important.

Also, starting without the hardest to reach kids helps reduce the risks of creating a new school.

I’ve seen new charter schools nearly collapse because of 5-10 kids who have severe mental health afflictions.

Of course, it’s not scalable for every school in a city to not serve the lowest performing 10% of students. And I also worry about segregating these students into separate schools.

But I think a reasonable policy would be to allow schools to get their footing and reach a bit of scale before having to build programs for the hardest to reach students.

 

Sometimes it doesn’t matter if it worked

Rand just release the initial evaluation of de Blasio’s Renewal School program.

The program cost $773 million.

The researchers found that the program did not improve student achievement.

The Renewal program has only been around for three years, so it feels a bit early to tell if it worked or not.

But I think “did it work?” is an important but secondary question to ask with a program like this.

The most important question is: “will it last?”

Even if the program had it gotten results, I’m skeptical that the program would have lasted.

The Renewal program had two major things going against it’s sustainability.

First, the program was deeply tied to de Blasio and the previous superintendent, Carmen Farina.

Second, there were no governance or legal protections for the program.

Taken together, this meant that the next mayor / superintendent would likely replace this initiative with their pet initiative.

This already had started to happen a year ago. When Farina stepped, the new superintendent, Richard Carranza, said the idea behind the Renewal program was “fuzzy.”

I viewed this as code for “it’s not my signature project.”

Admittedly, even I’m a bit surprised at how quickly the Renewal program collapsed. I thought it would at least live through the de Blasio administration.

We do live in a democracy (thankfully), so no government funded program, even if it works, is guaranteed to last forever.

But I would not have spent $773 million dollars on program that could be so easily undone.

I am such a deep believer in non-profit governance because it greatly increases the chance that something great can last.

In New Orleans, since the outset of the reforms, we’ve had three mayors and five superintendents.

And the work continues.

 

 

 

Minor feedback and reflections on Ray Dalio’s capitalism manifesto

Ray Dalio just published a two part manifesto on how capitalism needs to be reformed.

Dalio’s Diagnosis 

Dalio’s main point is that “capitalists don’t know how to divide the pie well and socialists don’t know how to growth the pie well.”

This is a problem for capitalist countries because rising inequality has both negative affects on individuals and countries: high inequality reduces economic mobility and weakens political governance.

I am not an expert in inequality, economic mobility, or governance, so it’s hard for me evaluate his specific claims.

But my instinct is too put a little less emphasis on capitalism.

Rising inequality seems to be the trend of almost all political systems.

Walter Schiedel’s book The Great Leveler does a good job of showing how inequality tends to rise in most societies (any system can be gamed) and that it’s often only war, revolutions, state collapse, and plagues that bring inequality back down.

Dalio thinks there’s something specifically inherent about capitalism that leads to inequality. I think there’s something generally inherent about governments and power.

Once humans left hunter gatherer societies, rising inequality has been the norm in most societies.

Capitalism is just the latest manifestation of the trend.

Stability vs. Growth 

Another argument which is somewhat implicit in Dalio’s argument but I wish would have been more explicit: given the incredible compounding of even small GDP increases, we should care much more about societal stability than maximizing growth.

Over the long-run, humans will be very rich by today’s standards so long as we can survive a couple more centuries.

In most cases, growth and stability go hand-in-hand, but when they don’t (perhaps in the case of inequality), we should err on the side of stability.

Can We Do Anything About It?

Schiedel’s book presents a bleak picture about a society’s ability to reduce inequality.

But humans haven’t been around for that long. We’ve never been this wealthy. And never before have so many humans lived in democratic societies.

It took us from the Industrial Revolution to the end of World War II to find a governmental system that fit our last major round technological and economic advancement.

Most advanced countries are now some form of capitalist democracies with large welfare states (at least by historical standards).

I hope our next governance search requires less bloodshed.

If it’s simply a matter of adjusting our current system, this might be possible.

If it’s a matter of finding whole new systems of governance, then that likely means wars and revolutions, which we should all fear given the weapons at our disposal.

Revisiting Diane Ravitch’s “A Challenge to KIPP”

7 years ago, Diane Ravitch wrote a blog post called “A Challenge to KIPP.”

In the piece, Diane accuses KIPP of cherry picking students and challenges them to serve an entire district.

When I gave my lecture, I chastised KIPP for encouraging the public perception that all charter schools are better than all public schools and for failing to denounce the growing numbers of incompetent, corrupt, and inept charter schools. I talked about the oft-heard complaint that KIPP cherry picks its students and has high attrition, which KIPP denies. I challenged KIPP to take over an entire inner city school district that was willing and show what it could do when no one was excluded.

She then makes the point in a more forceful way:

KIPP should find an impoverished district that is so desperate that it is willing to put all its students into KIPP’s care. Take them all: the children with disabilities, the children who don’t speak English, the children who are homeless, the children just released from the juvenile justice system,  the children who are angry and apathetic, and everyone else. No dumping. No selection. No cherry picking. Show us what you can do. Take them all.

This is part of what made New Orleans so important. KIPP didn’t grow to serve every kid in the city (which I don’t think would have been good, different kids thrive in different environments).

But charters did grow to serve every student in the city. And the last 20% of kids reached were harder to educate than the first 20% of kids reached.

But now, to use Diane’s words, charter schools in New Orleans “take them all.”

And they do it better than ever before.

Public charter schools in New Orleans have evolved to offer amazing programs for students with severe special needs. They also serve students in tough life circumstances, like teen mothers and those with extreme mental health and behavior diseases. None of these programs are perfect, but they are so much better than what New Orleans offered these students before, and some of these programs are on the path to be national exemplars.

And what about the results?

As readers of this blog know, New Orleans achieved some of the most impressive student achievement gains in our country’s recent history.

So is Diane Ravitch now saying: “New Orleans answered my challenge to KIPP. Charters enrolled all students and increased educational opportunity. I’m curious to learn more about how they did that. And I wonder if it could work in other cities?”

Of course not.

Unfortunately, with so many charter critics, the goalposts just continue to move.

But for the rest of us, we can keep the goalposts in place.

We can learn from successes like New Orleans, and we can try to figure out what’s scalable and what is not.

Beware of under powered studies

Mathematica just released a study on how charter middle schools impact college enrollment and completion. Before I dig into the study, some quick context.

The research base on the connection between test scores and life outcomes

An important research finding over the past decade has been that schools with neutral test score impacts sometimes achieve positive lifetime outcomes for their students.

Equally important, schools with positive test score impacts often don’t achieve positive life outcomes for their students.

Another important finding (a finding that is often glossed over by my libertarian leaning friends) is that schools with negative test score impacts are much less likely to achieve positive lifetime outcomes for their students.

Taken together, these findings have narrowed my belief in test based accountability. Previously, I put a lot of weight on the difference between positive, neutral, and negative test score schools.

Now I focus more attention on selectively transforming schools with negative test score impacts.

I also care much more about parent demand than I used to.

The Mathematica Study Limitations 

The Mathematica study has two very major limitations.

First, it covers charter schools from a previous era (2005-2007). In the subsequent decade, many charter schools (especially those we work closely with) have increased their focus on getting kids to and through college and into meaningful careers.

Second, the study is very small. The sample only included 10 urban charter schools. Additionally, only 3 schools in the entire study had poverty rates over 75%!

The small sample means the study is fairly under powered. With a sample this small, only large effects will be picked up. A modest but positive correlation between achievement and college graduation, for example, would not be statistically significant in this study. 

Other similar studies have looked at entire states, and included many more high poverty charter schools, so I place much less weight on this study.

Findings

The study did not find a statistically significant relationship between test score and college completion results. The chart below details the major findings, with each dot representing a school.

College Completion 

mathematica 2

When you eyeball the graph, some trends emerge, especially the low number of schools in the bottom right quadrant (high test effects, low college effects). 

When I emailed the authors about this trend, they said they were not statistically significant and we should be cautious to draw conclusions by looking at patterns the scatter plot. 

In Sum

The small sample size and the decade old study window period mean that the study itself won’t move my opinion on the issue that much.

Only 3 schools in the study were high poverty schools. Nearly all of the schools that we work with are high poverty schools.

In partnership with the Arnold Foundation, we’re supporting a bunch of more research on the issue, and I’m excited to learn more. 

I will change my mind if the findings change.