Category Archives: Decision Making

Rearranging how parents get information about schools may increase educational opportunity at no cost

Leave a reply

A .2 standard deviation increase in academic performance is a pretty good sized effect.

In this study, the .2 effect equates to attending a school that is 5 percentile points higher in ranking in academic performance.

Most interventions that achieve a .2 effect cost money.

This intervention costs nothing.

Unified Enrollment Systems 

Over the past decade many cities have adopted unified enrollment systems. These systems allow families to go online and view information on all public schools in their city, and then submit their ranked preference of schools to the government. New Orleans, Indianapolis, Denver, Chicago, Newark, Camden, and New York all have some version of this system for at least some grades.

These systems are great in that they give parents more information, allow them to easily apply to schools online, and help policy makers get information on which schools are most in demand by parents.

I’ve previously written about how the user interfaces for these systems diverge greatly in in quality. Some feel like you’re using a great iphone app and some are barely better than opening PDF files.

How Does User Interface Affect School Selection?

In this study, researchers worked with a consumer testing company to recruit a group of parents to use a generic unified enrollment system to select a school.

They then broke the sample into groups and presented a different user interface to each group, with the aim of testing how presenting information would impact school selection.

Here’s what they found:

Screen Shot 2018-11-07 at 8.27.24 AM

You can see the biggest effect (.19 standard deviations) is in the row “default sort order” and the column “distance.”

The researchers created two default sort orders: in the first case, you put in your address and then you are shown the schools nearest to your address; in the second case, you are shown the highest performing schools available to you.

The researchers found that if you make academic performance the default sort order, parents ended up picking schools that were +5 percentile points higher ranked on academic performance.

Making academic performance the default order costs no money.

Limitations 

The study has some real limitations.

First, the stakes weren’t real. The parents weren’t actually selecting a school that they would send their child to. This probably meant they put less effort into the school selection. They also weren’t able to get information from other sources (like friends and family).

Second, having parents pick schools with higher academic performance ratings is only useful if those ratings accurately measure student learning. In cities that use value-added methods for school rankings, I’d feel more comfortable with this nudge. In cities that mostly use absolute test scores, I’d feel less comfortable.

Third, academic performance isn’t everything, and parents select schools for a variety of reasons. Ultimately, designers of the interface do have to make choices, so I’m ok with a bit of nudge toward academic performance, but I don’t think we should make this nudge at the expense of giving families a wholistic picture of schools.

All that being said, the study shows that small and easy to make changes in user interface may have an impact on how families select schools.

If I was a government official that managed a unified enrollment system, this study would lead me to experiment with similar interventions for my own city’s system.

At the very least, I’d want to make sure that my user interface decisions were deliberate and values based rather than ad-hoc and random.

Access and Supply

Lastly, great unified enrollment systems are about equalizing access to great public schools.

They do nothing to increase the number of great public schools.

Cities would also do well to do all they can to help their best public schools expand.

The problem with “risk / mitigation” sections in business plans

Leave a reply

I see a lot of business plans and our team writes a lot of strategy memos.

I’ve always been a little frustrated with “risk / mitigation” sections of these documents, including some I’ve written myself.

“Risk / mitigation” sections feel like an exercise in confirmation bias. They almost always affirm the correctness of the business plan’s thesis.

The logic path of a risk / mitigation section is: “here’s our idea, there are some risks, but here’s why we’re going to do our idea anyway.”

This is a modestly useful exercise. It’s worth understanding the risks of a plan and preparing for them. But this type of thinking does not force you to analyze the “crux” of your plan – i.e, what are the major trade-offs you’re making, and under what future conditions could you be proven wrong.

I prefer to see both of these lines of thinking in a business plan: first, a clear identification of the trade-offs in any major decision, and, second, a statement of what might make you change your mind about your plan’s thesis.

Any very hard decision will include difficult trade-offs. And any hard decision might also result in you being wrong. Identifying these ahead of time will help you be mentally prepared to make difficult shifts down the road.

It will also decrease the probability that you drink your own kool aid.  There is a humbling effect in writing down potential future conditions that will indicate that you are near failing or have already failed.

I hope that one day every business plan includes “trade-off” and “we will know we were wrong if” sections.

Doubling down when you’ve been wrong

2 Replies

From 2012 to 2016, three reforms took place in Memphis: the Gates Foundation teacher evaluation reform, the Achievement School District (ASD) charter school turnaround effort, and the district run izone turnaround effort.

In private conversations, I predicted that neither the teacher evaluation nor izone efforts would work. I thought the ASD charter effort work would work. The Arnold Foundation funded research to help us understand if our beliefs about how to help children were correct.

Five years later, the results paint a different picture than my predictions: the teacher evaluation work did not improve achievement. The ASD charters have yet to deliver results. And the izone schools have done the best.

I was wrong about the izone and the ASD.

So what to do now?

The short answer: expand non-profit governance to protect the gains of izone schools, and continue to grow the best non-profit public charters and replace the worst.

The longer answer is below.

Responding to Being Wrong

There are three ways to respond to being wrong: ignore that you were wrong, change your behavior, or admit that you were wrong but double down.

Ignoring that you’re wrong is never good.

Changing your behavior is often the most reasonable response.

Doubling down comes with substantive and reputation risk, but it can be the right thing to do.

Memphis Results 

The teacher evaluation reforms went roughly as I expected, so I won’t discuss that too much here, other than saying that my predictions were based on the belief that execution would hamper implementation in big urban districts, which seems to be exactly what happened.

Here are the results of the izone and ASD charter efforts:

 

 

This slideshow requires JavaScript.

The izone had a positive .2 stand deviation effect over five years. This is fairly good impact and it’s great to see these schools helping students.

The ASD schools had no impact.

What I Got Wrong

I made a few mistakes.

First, I underestimated the school district’s strategic and executional competency. In creating the izone, the district successfully recruited a lot of their best educators to their izone schools, and then appointed a talented leader, Sharon Griffith, to oversee these schools. This strategy (concentrating talent) and execution (selecting a great leader) worked in ways I did not expect.

Second, I overestimated the abilities of charter networks to deliver results in the first few years of the turnaround efforts. My assumption was that expert charter leaders would be able to deliver better results pretty quickly, even though they were going into very difficult situations and were often coming from out of town. Unfortunately, a few operators really struggled in the early years, with one of them, Yes College Prep, choosing to not even open up a school (this was a big hit as YES is considered to be one of the best networks in the country).

What I Would Have Done Differently 

I should have realized that great educators in the traditional system could achieve good results if given autonomy in a competitive environment. I have changed my beliefs on this issue based of the work of the izone, the work in Lawrence, Massachusetts, and my lived experience of seeing great teachers and leaders in all types of schools.

Rather than predict the izone wouldn’t work, I should have predicted that the izone might work, but advocated for non-profit governance over the zone so that it could sustain its gains. This remains my worry with the izone: that it will not be able to sustain its impacts over the next 5-10 years due to its governance structure, which makes it vulnerable to political shifts.

I also worry that it can’t really scale. Putting the best educators in the worst school is more of a one time intervention rather than a strategy that can grow over time, and it has some negative effects on the schools that lose their great educators.

Second, I should have realized that early stage charter operators (either newly started or new to Memphis) would struggle to scale through full school turnarounds in a difficult political environment. In hindsight, I should have advocated for more one grade at a time roll outs of newer operators. While this would have been more disruptive to families at the outset, it would have been better for kids over the long run.

What to Do Now

If the above diagnosis is true, it doesn’t really call for a dramatic change in strategy.

Rather, it calls for trying to get non-profit governance over the best district schools, growing the charters that are working, replacing those that aren’t, and starting new non-profit district and charter schools under more favorable conditions.

Getting non-profit governance to the best district educators would (I think) require a change in legislation.

Growing the best charter schools and closing the worst  can be done under current policy.

Evidence to Support Doubling Down

Researchers have found evidence of charter school sectors improving over time. This paper on charters in Texas found significant sector improvements, which led the researchers to write:

The findings suggest the value of taking a longer-term perspective when evaluating the impact of a major educational reform such as the introduction of charter schools, especially when the success of the reform ostensibly depends on parental decisions and market forces.

Additionally, previous research on Memphis charters found strong positive effects:

Screen Shot 2018-06-23 at 8.03.44 AM

These results mirror the results of many other urban charter sectors, which generally find positive effects.

So there is a research based case for predicting that the Memphis charters will improve over the next 5-10 years. We have already seen this with some operators, such as Aspire.

In Sum 

The research has increased my belief that great district school educators, with autonomy and support (and often pressure on local districts from the state to give this autonomy) can really help struggling students.

I still believe that these gains risk backsliding due to political shifts, so I support giving these schools non-profit governance so they can build enduring organizations (both Denver and Indianapolis have mechanisms to give district schools non-profit status).

I also still believe that growing the best charters and closing the worst is one of the best hopes for improving student achievement in Memphis.

So, in many ways, I’m choosing to double down despite being wrong, with the caveat of calling for non-profit governance for more types of public schools.

While this is uncomfortable, I don’t know a better way forward. I think doubling down will help children.

I understand that many of those who I disagree with say “we just need more time.” And I’m making a version of this argument here. So I’ve tried to lay out all my assumptions as clearly as possible, both for the sake of transparency, as well as with the hope that these assumptions can be corrected if they are wrong.

If you support neighborhood schools you also (unintentionally) support segregated schools

6 Replies

A reminder: for the foreseeable future, supporting neighborhood schools means de facto supporting segregated schools.

The reason is obvious: neighborhoods in our country are highly segregated.

I think our country would be better if our neighborhoods weren’t segregated, but I don’t think that’s going to change anytime soon.

So if every kid goes to their neighborhood school, we will have segregated schools.

There are a couple ways out of this.

We could restructure enrollment and bussing rules to avoid segregation, but this would mean that a lot families would have to attend schools outside of their neighborhoods.

And  I don’t think it’s the rich families that are going to send their children to poorer neighborhoods.

So we’ll need to be bussing in low income students into rich neighborhoods. This might be the right thing to do, but that means a lot of low income students won’t be attending their neighborhood schools.

It also means we’ll likely have a lot of white flight, which, while unfortunate, is neither good for integration nor a city’s tax base.

The other option is to create all choice systems and allow schools to preference students in a way that increases socioeconomic integration (this could be done through a unified enrollment algorithm).

For this strategy to be successful, schools will have to proactively create enrollment rules that increased integration, and families will have to proactively choose schools with this mission.

This will obviously be a slower process than forced integration via non-choice bussing systems.

But I think it will be much more durable.

Ultimately, you can’t force people to integrate our current version of segregated schools if they don’t want to. They will either move or kick out the superintendent who forces it. As a country, we rightfully changed the laws that forced segregation, but we’re still left with the fact that many people don’t really want integration, at least not if it involves any bit of giving up of privilege.

So, no, you can’t force integration. But you can give educators the opportunity to say that their school will prioritize integration. And you can make it easier for families to choose these schools.

The road to school integration is not through neighborhood schools. And it’s not through forced enrollment patterns.

The road to school integration is through people actually wanting it, and for government to create open systems that allow these desires to be actualized.

If you do nothing, people will attend their segregated neighborhood schools.

If you force it, they will flee.

If you build it, they might come.

The Current Brutal Reality of Education Reform and Wage Growth

10 Replies

edu wealth

Going from 16K to 18K in Annual Wages 

Last week, I did a post on Fryer and Debbie’s excellent new study on the Texas charter sector.

I emailed the authors about my hypothesis that the growth of high-quality charters – even if they aren’t that much better than average traditional schools – could still be of great value if these new charters displaced chronically failing schools.

Roland was kind enough to respond but pushed that even if my hypothesis is true, the story still might be a depressing one.

His point: the numbers from his study indicate that even if we replaced all these failing schools with high-performing charters, we’re still only talking about ~1-2K in extra earnings per year for these students.

Given that many of these students end up lower income brackets, this might mean going from 16K to 18K a year in annual salary. Hardly game changing in terms of life outcomes – and surely not a ticket to the middle class.

Confronting this Potential Reality

When a study tells you what you don’t want to hear, the first reaction is often to not deal with it (in some ways I did this in my previous post).

So everyone in education reform needs to deal with this potential reality: there is some possibility that the best that education reform has to offer can only, on average, move a student from 16K to 18K a year.

Of course, this is only one study of one state. We don’t yet know if these numbers will hold under different contexts, methodologies, or timeframes.

But, at the very least, your belief that a great school can radically increase wages should be a little lower after reading this study.

Other Considerations

I’m still mulling this over, but in conversations with Roland and folks I work with, certain ideas bubbled up:

The data doesn’t capture recent improvements: A lot of the best charters have only really started focusing on college and career over the past 5 years or so. As such, the students who received the full suite of redesigned high schools, counseling, and career support aren’t represented in this study. To the extent you believe the best charters are problem solving machines, you might believe this to be true.

The work is generational: Perhaps reformed schools can only, on average, push students who would have been in deep poverty to achieve average poverty / lower middle-class status. And perhaps their children, who will grow up in better educated environments, will the be ones  to more fully make it into the middle class. But this story could be unwound through raised expectations: if we told kids they were going to make it to the middle class, and they don’t, how will they react?

Colleges are the bottleneck: Perhaps these real gains in learning are being wasted by ineffective two year and four year colleges – and that without higher education reform we won’t be able to translate K-12 gains into wage increases.

Society is tough: Just because you’re better educated doesn’t mean you can overcome racism, lack of social capital, and an over-reliance on signaling.

More interventions are needed: Great schools can’t solve everything; interventions that work on family poverty, health, and parenting are needed for schools to really move kids as far as they need to be moved.

The schools aren’t really that good: A bunch of teaching to the test just jacks up crystallized knowledge but doesn’t really give kids the human capital qualities they need to succeed in the workforce.

What Do you Do in the Face of Ambiguity?

Leaders need to make hard decisions in the face of incomplete data.

Often times, this means relying on some combination of probabilistic thinking, intuition, ideology, and philosophy.

But, at some point, you need to walk away if the data is telling you what you’re doing is not working.

I don’t think one study is enough to walk away from the promise of urban charter schools, especially since they’ve achieved so much on less penultimate markers.  I think there’s a lot more experimentation and research that needs to be done to help us understand if we can translate academic gains into wage growth.

But it’s worth thinking about when you would walk away.

Because if there is no point at which you’d walk away, then what do you really stand for?

The Answer is 6.7 Miles. What is the Question?

2 Replies

Screen Shot 2016-06-15 at 8.01.52 PM

The question is: how far, on average, would a family send their child to attend a school that is in the highest category of the state accountability system compared to a school in the lowest category of the state accountability system?

This is from a recent report on the DC public school system. The analysis, while useful, isn’t perfect in that it only includes families who utilized the enrollment system, but it does add to the emerging literature on the revealed preferences of families that participate in transparent enrollment systems.

 

___

Here’s another answer: it increases racial integration.

The question is: does DC’s unified enrollment system increase or decrease racial segregation?

Screen Shot 2016-06-15 at 8.06.39 PM

Shockingly enough, assigning families to neighborhood schools that are zoned by property values is not a great way to decrease segregation.

 

___

Answer: Unclear.

Question: Do parents care about a school’s academic growth (as opposed to absolute test scores)?

Screen Shot 2016-06-15 at 8.20.46 PM

Interesting but not shocking. Parents probably care a lot about peers and status.

Also interesting, this seems more true of low-income families:

Screen Shot 2016-06-15 at 8.25.26 PM

This raises an interesting question for policy makers: given that growth more accurately measures a school’s impact, should they design grading systems that prioritize growth (as DC’s charter framework does) even though low-income parents might care more about absolute scores?

Or perhaps not – maybe low-income families aren’t considering the growth based performance framework because the government is hiding this information:

Screen Shot 2016-06-15 at 8.27.05 PM

___

One last answer: Families who aren’t assigned to a school in the lowest performance category, as well as the politicians and superintendents who seek their favor.

The question: who loves neighborhood schools?

It remains shocking to me that public leaders in cities such as Oakland are vehemently opposed to unified enrollment on the grounds that such systems will undermine public education.

The only thing a unified enrollment system undermines is the privilege of those who benefit from institutional racism and widespread income inequality.

 

 

Is No Excuses or Personalized Learning the Low Hanging Fruit of School Improvement?

3 Replies

On average, I visit a school every other week or so. For the most part, these schools are equal to or better performing than the median urban district school.

During these visits, one question I usually mull over is this: if I was leading the school, what would I focus on to drive the next phase of improvement?

Often times, what the school leader is focusing on and what I would focus on are at odds.

I don’t have extremely high confidence in my analysis, so consider the below speculative.

____

Here are the things I most often here school leaders saying they need to improve on: personalization, student ownership, and critical thinking. Tactically speaking, this often leads them to experiment with new models of instruction and technology.

All good things.

But I’m often thinking that the school really needs to get better at: instructional delivery, higher ratios of student intellectual engagement, and more effective use of small group instruction.

Most school leaders seem to believe that they have the basics down and need to go from good to great.

I tend to think that most schools are mediocre at the basics of things such as cold calling, wait time, efficient time on task, and tutoring – and the other hall marks of the no excuses model.

___

So here’s some questions on my mind:

  1. Should the median charter school be focusing on getting better at the basics of the no excuses model or experimenting with deeper innovation?
  2. If it’s true that the median charter school is still mediocre at the no excuses basics, what should we take from this? That high fidelity to the no excuses basics is operationally hard to scale for either intellectual, emotional, or human capital reasons? That many leaders don’t think the no excuses basics work?
  3. Is there a progression of improvement (i.e., you need to get the basics right before you work on deeper innovation) – or does shifting to more innovative models allow you to bypass the no excuses basics and still get academic gains?

___

My hunch is this: only the top tier charter organizations and the very best entrepreneurs should be working deeply on the margin of innovation.

Most charter schools should be working on the margin of better adoption of the tenets of the no excuses model.

Once new models are hammered out and refined – and get better results than the no excuses model – then the median charter school should begin adopting these new models.

But not before that.

In sum, I think better fidelity to the no excuses model is the low-hanging fruit of school improvement.

Maybe I’m wrong? Maybe I’m very wrong?

The Challenge of Separating Emotional and Intellectual Agreeableness

2 Replies

There is a decent amount of research showing that agreeableness (as measured by the five factor personality test) is not always associated with strong professional outcomes.

Specifically, agreeableness can reduce results orientation and create opportunities to be taken advantage of by colleagues who better use power to achieve their desired ends.

That being said, agreeableness need not be all bad: to the extent that it helps cultivate large, loose networks, agreeableness is likely of use to leaders in attracting talent and coalition members, especially in the non-profit sector.

___

Many times, I’ll be in a conversation with a colleague, grantee, or potential grantee and there will be a small war going on my head: part of me wants to nod my head, smile, and ask probing but pleasant questions – while another party of me wants to dig in very hard on everything that might be wrong about what we’re discussing.

I have a strong desire to be both emotionally agreeable and intellectually disagreeable.

Which begs the question: is it possible to be emotionally agreeable while being intellectually disagreeable?

___

I’m not sure. But here’s some things I try to do:

  • Utilize processes that create a safe space for intellectual aggression (i.e., assigning someone to be the devil’s advocate in a meeting).
  • Using hedging phrases such as “I might have this wrong, but….” that soften the blows of intellectual aggression.
  • Trying to separate my empathy for a person with my disagreement with her ideas – so that my intellectual disagreeableness does not bleed into full blown personal animosity.

If you have any other tools, let me know.

I struggle to get the balance right.

Sometimes I feel like I’m too agreeable, and sometimes I feel like I’m too intellectually aggressive.

I Think High Schools Should Teach More of These Two Things

2 Replies

Trying to answer of what, on the margin, high schools should teach more of gets at the heart of numerous subquestions, including:

  • What is the role of schooling?
  • Where is the economy heading?
  • What level of content can high schoolers handle?
  • What is already being covered well?

I will steal from a bunch of bloggers, academics, and practitioners in a way that will be a little hard to cite, but it’s fair to say that little that follows below is that original.

A Primary Goal of Schooling is to Normalize that which is Not Normal

In hunter and gather societies, Algebra is not normal. Nor is conscientiousness. Nor is abstract logic.

You get the idea. As societies advance, there’s a bunch of things that may need to be taught because physical and cultural evolution do not provide enough guiding instruction.

Of course, school is isn’t the only way to learn things that aren’t normal, but we’ve chose to make it a major source of normalization.

So one way to answer the question – what should high schools teach more of – is to consider what is not normal now but that we would wish to be normal in the future.

Closely related: what is close enough to normal now that schooling could provide a meaningful nudge to normalization?

What Would Make the World Better if It Were Normal? 

A lot of recent books and blog posts have influenced me on this, perhaps most notably Superforecasters,  Scott Alexander’s post on how hard things are for a lot of people, and Triggers.

The thesis of each of these sources are as follows:

  1. Superforecasters = always start with baseline research and data before making any decision.
  2. Scott Alexander = even in America, a lot of people are suffering with major issues such as chronic pain, drug abuse, and unemployment.
  3. Triggers = adult behavior change is very hard but possible.

Over the past few years, recent events in my own life have driven home the importance and relevance of these theses.

What would make the world better if it were more normal?

I think these two things:

  1. An increased internalization that opinions should be formed based on baseline data and research. I mean this both in the policy sense (should we raise taxes?), the business sense (which website design should we use?), and personal life (which nursing home should I use?).
  2. Adult behavior change is possible and specific techniques can increase the chance of success. I mean this in the professional sense (I need to ask more questions in meetings), the interpersonal sense (I need to talk less about myself in groups), and the spiritual sense (I need to meditate more frequently).

Can High Schoolers Handle This?

I don’t know, but my guess is yes. Research trials could tell us. But none of the above are intellectually taxing in terms of complexity.

Rather, these two learnings are more just specific applications of conscientiousness, humility, and growth mindset.

It’s my strong guess that strong marginal improvement could be made in these areas without students ever entering a college campus. It feels like high school material, but I might be wrong.

Providing the Cultural Pressure and Intellectual Tools to Achieve Normalization 

With a few exceptions, high school curriculum covers a lot of important material. My guess is that, on the margin, more traditional content is not what is needed (i.e., more advanced math).

Rather, at the margin, I think there may be a lot of gains to be made in providing cultural pressure – and giving student the intellectual tools – to normalize the tools of data usage and adult behavior change.

But all this is surely speculative, and I don’t have strong confidence that I’m right.

Perhaps some high schools might push in this direction and see if it makes a difference in students’ lives.

Book Review(s): 6 Books on Our Mental Limits

Leave a reply

I’ve had some good reading time over the past two months and have been able to get through six books (as well as the new Dragon Tattoo book, which will not be reviewed here):

  1.  (adult behavior change)
  2.  (predictions)
  3.  (Google’s HR systems)
  4.  (utilizing simple rules to guide decisions)
  5.  (evolution as a principle for all change)
  6.  (how national IQ is more important than individual IQ)

All are worth reading.

Here are some major themes that ran through them all:

We Have Weak Minds

Triggers pushes hard on how much environment impacts us.

Super forecasting details how badly pundits do at prediction because they rely on situational judgment rather than baseline data.

Simple Rules makes a convincing case that the world is too complex to navigate by fully analyzing every situation.

The Evolution of Everything rightly argues that even our geniuses are most often well situated for breakthroughs due to past intellectual evolution, not because they along were capable of achieving such breakthroughs.

Collectively, We Have Better Minds

Hive Mind demonstrates how individual minds are made more effective by having other good minds around.

Triggers lays out an accountability regime whereby other people hold you accountable for your behavior commitments.

Superforecasting talks about how even the best forecasters improve when working together.

The Evolution of Everything narrates how it is our collective knowledge, built over the ages, that allows to enjoy the fruits of modernity.

Those Who Use Data Effectively Will Win 

Work Rules! vividly portrayed how heavily Google relies on data analysis to make any decision, be it about people or anything else.

One memorable quote went something like a manager saying this: “If you don’t give me data, I will give you my opinions, and you don’t want that.”

Super forecasting is all about how baseline data is needed to anchor any situational judgment.

Triggers recommends systematic daily tracking of any desired behavioral change.

How I’ve Changed Because of these Books

  1. After reading Triggers, I created an end of the day 10 question checklist to hold myself accountable for the behaviors I’m trying to implement (I use an app to record them every night).
  2. After reading Superforecasting, I’ve tried to ensure that we conduct a  research review of any issue before even beginning to make judgments, to ensure we understand baseline data.
  3. After reading Work Rules! I reflected on how much I over relied on my own judgment when I led NSNO. I should have done a better job of always asking for the data before making any managerial decisions.
  4. After reading Simple Rules, I revised a decision checklist I had made for grant making to include a priority rule (most of them were boundary rules and stop rules).
  5. After reading Hive Mind, I reflected on my strong preference for very open immigration. While I still hold this belief, the book helped me understand where and why I might draw limits.

I don’t know if I will be successful in sustaining any of these behavior changes. But I hope I can.

If you have a chance, I recommend picking any of the books up for holiday reading.