How I learned to stop worrying and love the data

I have always been comforted by data. Just as I don’t think I will win the lottery, I also don’t think I will be involved in a plane crash. And now, as I have spent the past three years filling my head with mathematics and statistics, I also have become more prone to go ahead and run the numbers on how much I should worry about certain things.

Emily Oster, associate professor of economics over at U of C felt the same way as a pregnant future mother. She turned her obsession with getting to the bottom of all the dozens of “pregnancy rules” doled out (inconsistently) by OB/GYNs, doctors (and midwives) into a book (read the intro here). In it she finds that many of them are based on highly flawed or outdated studies, or that the rules that they foster are the strictest possible interpretation of less-than-clear study results. I wish we had had it available when our daughter was gestating. It would have allowed my wife to enjoy her day to day life as a pregnant woman a great deal more.

The book was critiqued in reviews like this one purporting to “debunk” it (by comparisons between Oster and anti-vaccination “activist” Jenny McCarthy. The author here comes down firmly on the side of listening to whatever often conflicting advice your doctor(s) give(s) out citing “Dr. Marsha McCormick, a professor of maternal and child health at Harvard Medical School and the Harvard School of Public Health, who can’t, for example, imagine a medical doctor evaluating economic data and dispensing policy recommendations or financial regulations for the housing industry.” This may be good advice for Dr. McCormick and other medical professionals to follow. A 1978 study in the New England Journal of Medicine found that 87% of a random sample of physicians, students and residents serving in hospitals of the Harvard Medical School could not answer the following question correctly:

If a test to detect a disease whose prevelance is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease, assuming you know nothing about the person’s symptoms or signs?

If you are interested in how to answer such a question, which is a straightforward application of Bayes’ Theorem, go here and see how you do in getting there. The correct answer is that a positive result indicates a 1/51 (or approx 2%) chance of having the disease. Most answers from respondents in the study ranged from 95% likelihood of having the disease down to around 50%. Even this lower estimate overstates the risk posed by a positive test by 25 TIMES the actual risk! The authors’ conclusion was that “in this group of students and physicians, formal decision analysis was almost entirely unknown and even common-sense reasoning about the interpretation of laboratory data was uncommon.(Such errors have been replicated in numerous studies over many years up to the present day. Here is another more recent study of the same type without a paywall.)

While this sort of analysis is not super easy for a lay person, it really should be pretty manageable for a trained physician who makes diagnostic decisions based on the statistical data contained in medical studies. It is apparently not for many or even most physicians. But such analysis is very straightforward for any trained economist. Bayes’ theorem is covered in virtually any intermediate-level undergraduate economics course and again in any introductory or mid-level statistics or probability courses. By the time one completes an Econ PhD, simple applications of Bayes’ Theroem have been beaten into a students head for years and years. So it may well be that Emily Oster may be able to interpret the results of clinical studies in a manner that physicians should take note of based on the extent of her mathematical training alone. If we venture into the realm of potential problems with study design, discerning between correlation and causation, and other such methodological analysis of medical studies, then an economist has a clear advantage over most practicing MDs.

Another point that Oster makes in the intro above is that in economics the focus is on weighing trade-offs. Since most people don’t do things habitually that have no personal value to them at all, if someone prefers to do something it usually indicates that they gain some enjoyment or value (let’s follow tradition and call it “utility” henceforth) from doing so. We all seek utility from the things we do and we are constrained, by time, money, information and many other factors, from doing them exactly as we would prefer to. In pursuit of utility we all make trade offs with our health and our very lives every day. We eat too many sweets or we cross the street while texting or we smoke or we have a couple of drinks and figure it’s okay to drive home because the alternative is terribly inconvenient or we stay on the road when sleepy or we take our medicine in different doses than prescribed and so on. In every case, we are making a decision about risks and rewards. In most of these cases we don’t even have good data to guide us, just a gut feeling. We also worry too much about very, very low risk situations like the risk of an airplane accident or the risk of having our homes broken into.

Which is all a long way of introducing a trade-off I have been weighing in my own life. It is about the seemingly arcane decision of which way to face my 16 month old daughter’s car seat in our car. Flashback to pre-2002. The American Academy of Pediatrics (AAP) recommended at this time that children should face backwards in a car seat until they weighed 20+ lbs with good neck muscle strength, which tended to correlate with turning one year old. So the heuristic used by many was to say that you can face your child forward after they turn one year old. Then apropos of this 2007 paper “Car safety seats for children: rear facing for best protection” by Henary et al., the recommendations were revised to say that children should stay facing backwards in the back seat until they reach either age two or the limit of their rear facing car seat (which in infant to toddler seats is usually about 40 lbs).

If you search this sort of thing on the internet you are likely to come across a lay-person AAP press release such as this one. The meat and potatoes are as follows:

While the rate of deaths in motor vehicle crashes in children under age 16 has decreased substantially – dropping 45 percent between 1997 and 2009 – it is still the leading cause of death for children ages 4 and older. Counting children and teens up to age 21, there are more than 5,000 deaths each year. Fatalities are just the tip of the iceberg; for every fatality, roughly 18 children are hospitalized and more than 400 are injured seriously enough to require medical treatment.

New research has found children are safer in rear-facing car seats. A 2007 study in the journal Injury Prevention showed that children under age 2 are 75 percent less likely to die or be severely injured in a crash if they are riding rear-facing. (My bold)

Now, the first paragraph above is kind of strange since it refers to children age 4 and up. This has nothing to do with children between the ages of one and two years. Also 5000 deaths per year in the US for children and teens up to age 21 sounds like the results of a very low probability event, and that is for everyone from birth to drinking age. Then from these unrelated statistics we suddenly jump to the report’s finding that children under 2 are 75% less likely to die or be severely injured if rear-facing. 75% sounds like a lot right? But 75% of what? The answer turns out to be 75% of children under the age of two who suffered severe injury or death in a car crash and who happened to be sampled in the National Automotive Sampling System Crashworthiness Database between the years of 1988 and 2003 (I will get to problems with this sample shortly). The habit of making claims like “children under 2 are 75% less likely to die or be severely injured in a crash…” Is sufficiently misleading to even doctors, let alone the general public, that it warranted a recent blog post on the website of the Journal of the American Medical Association highlighting the difference between relative risk ratios (what the authors of the car seat study tout as their finding) and absolute risk reduction (what I am about to explore below). Physicians are implored to understand the importance of this distinction, which would imply that many currently do not.

So I wondered what was the probability on average of my family getting into an accident that fit this set of criteria? One absolutely critical thing to understand is that your average risk of being in a crash is based on miles driven, if you don’t drive, you are not going to get in a car crash. If you drive very little it is much less likely you will be in a car crash and so on.

Using the State of Illinois crash statistics for 2012, here is a walk through a number of statistical steps to a very conservative approximation (meaning overstating the risk at each step*) of the probability of my child being severely injured or killed based on the direction I face her car seat, given my family’s driving profile.

(To minimize monotony henceforth, I will state here that all of these figures are average risk, based on annual statewide statistics, so that I can dispense with adding “on average” to every sentence I write. I would characterize my wife and I as pretty average drivers. We have each been involved in a separate, minor, non-injury, other-driver-was-at-fault, accident once in the last decade or so.)

1) Probabilty of being in a car crash: 1 in 2.6 million (1 / 2,600,000) per mile driven.

We drive with our daughter an average of about 200 miles per month (a generous overestimate). This works out to 2400 miles per year. So our most basic probability of being in car crash with our daughter is 1 in 1083 annually. But there’s also this:

2) Proportion of crashes between the hours of 8 AM and 8 PM: 72%

This time period contains nearly 100% of the time we ever drive with our child in the car and serves to multiply the denominator of the above probability by 1.385. Thus our average probability of being in a car crash with any injury is closer to 1 in 1505 annually. (Edited based on feedback about an error.While the distribution of miles driven by time is hard to find, the National Household Transit Survey gives time breakdowns that don’t exactly match these hours but suggest that perhaps 85% of driving is done between 8am and 8pm. This would give a multiplier of  (.72/.85) and moves the probability of a crash to about 1 in 1279 annually. (Figures below have been edited to reflect this change.)

I will plug this average probability of being in a any sort of crash with our daughter in the car into the data contained in the report cited by the AAP guidelines. This study looked at a sample of car crashes with children in the car drawn from the time period 1988-2003. This is an unorthodox (actually an invalid) method of creating a sample based on the way it is subsequently analyzed, but I will take the methodology as valid for the moment. I will split the probability into two cases, rear-facing car seat (RCFS) and front-facing car seat (FFCS). The authors of this study use the Maximum Abbreviated Injury Score (MAIS) system to rank injury and death. In this scale 0 is no injury, 6 is death, and 2 is “moderate injury”. The study uses greater than or equal to moderate injury as the baseline measure they call “moderate and severe injuries” (even though the study discusses the risk of severe injury or death as the finding of interest).

3) Proportion of children in crash sample with no injury or light injury: RCFS = 99.5%, FFCS = 98.9%.

This translates the 1 in 1505 figure for my family above into the following two ratios. The absolute probability of severe injury or death (as defined in the study) for my child in an RCFS is 1 in 255,800 annually. The same probability with a front facing car seat is 1 in 116,272 per year.

Now, that seems quite low to me, but if it doesn’t seem low enough, let’s stop assuming that the research design is valid (because it isn’t). Keep in mind two things as we dig in here. One, I am a pre-first year graduate student in economics as I write this and, two, that these criticisms are quite obvious and valid in spite of my “pre-economist” status. While most readers probably will or will not take my word for this instead of going through the trouble of researching the claims and links below, I am happy to correspond with any readers who may like to press me further on them.

The paper in question (Henary et al.) uses data collected from the National Automotive Sampling System Crashworthiness Data System (NASS-CDS) on children 2 years and under involved in crashes between the years 1988 and 2003. From this data they obtain an initial sample of 1870 crashes. From these they exclude unrestrained children and also explicitly improperly restrained children, which excludes 29% of the sample. They also exclude children with unknown restraint use/type, which excludes another 23% of the sample. After these exclusions, they end up with 870 crashes to analyze.

There are already problems at this point, so let’s dig in. In doing statistical research and analysis, there are two primary types of data sets. The first is cross-sectional data. This is a set of data taken from one point in time. Examples might include data from the 2000 census or retail sales for Black Friday 2013. Such data sets look at some (often large) number of people, sales, car crashes (or what have you) at one particular point in time. These data sets can be used to analyze differences in outcomes between observations given the conditions that prevailed at the time of the sample.

The second primary type of data set is time-series data. This is data looking at the same set of subjects at different points in time. Examples of such data sets are changes in the value of the stocks in the Dow Jones Industrial index between 1990 and 2000 or changes in US automotive fatalities between 1988 and 2003.

Hopefully that last example gives a clue as to a major, major problem with the data set used in the Henary paper. This sample has been created by aggregating discrete events across many years and analyzing them as if they all happened at one point in time (in other words, they are collecting time-series data into a cross-sectional data set…this is a no no). It took the authors 15 years to collect together 870 crashes they considered useful to look at. (They then apply a statistical weighting based on a single year’s accident trends to claim the sample represents a nationally representative population sample.) By discarding time variables from their observations they are implying that cars, car seats, aggregate fatalities and any other thing you might care to wonder about did not change between 1988 and 2003 (and in fact they do not address this issue even in passing in the report). Let’s unpack the implications of doing this in a few parts.

First aggregate fatalities. Between 1988 and 2003 total fatalities declined from 3.44 per million miles driven to 1.48 per million miles. That represents a decline of more than 2/3rds over the time period that has been engineered to be a single period. The authors came up with a total of 42 infant car seat deaths over these 15 years (an average of around 3 per year) but we have no information about what years these deaths took place. Probability suggests that more of them came from the late 80s/early 90s than from the early 2000s. Look at the car seat samples below and think about whether this information would matter.

Now cars. It was the advent of the passenger seat airbag that resulted in the recommendation of putting car seats only in the rear of a car. During the period the sample was drawn from, the average age of the US auto fleet hovered around 10 years old so, among cars in this sample, airbags went from being non-existent to being an option for the driver’s side, to being an option for the passenger’s side, to being standard for both front seats in all cars, to including standard rear seat airbags. Would that seem to matter? Many of the front-facing deaths in this sample may have occurred in the front seat of the car as the standard protocols of car seat placement were only just being formulated during the earlier years of this sampled period. Airbags may have played a role in a number of these deaths as well.

And what of child restraint systems (CRS)? Virtually all the norms of car seat installation, operation and placement took place during this time as well, meaning that in many cases there were no established recommendations as to how and where to install quite primitive car seats. To give a sense of the dramatic differences in car seat safety knowledge around the sampled period, here are a few excerpts from this timeline of the history of child restraint systems.

1987: Survey of CRS use in 19 cities across the country shows 80% usage (correct and incorrect) of restraints for children under age 5.

1989: Evidence of CRS misuse grows; car seat inspection clinics in Virginia and California find high levels (87-93%) of errors.

1991: Almost all states have passed safety belt use laws. Many do not cover rear seat occupants and can be enforced only if the driver is stopped for another violation.

1993: CDC issues a the first public health warning on interaction between air bags and rear- facing child restraints (MMWR, Vol 42/No 4, April 16, 1993)

1995: July: First death of infant from being struck by passenger air bag while riding in the front seat in a rear-facing restraint. (First infant known to have been injured by air bag, November, 1994.)

1999: September: The tether part of the universal child restraints anchorage standard (LATCH) began with the requirement that all forward-facing CRs must pass a reduced head excursion test, for which almost all employ a tether strap.

2000: September: Phase-in of the tether anchor requirement of the universal child restraints anchorage standard (LATCH) continues, with 100% of all model-year 2001 passenger vehicles (including SUVs, pickup trucks, or vans) being required to have tether anchors.

So in the early part of the sampled period, as many as 90% of the approximately 80% (a high estimate it appears) who used CRS at all were probably doing so incorrectly. Then for years, many people placed children in the front seat with armed airbags. Also the entire modern system of anchoring for child seats was conceived of during this period and was only full implemented in the last year or two of the sample period.

Finally as to the seats themselves, the period 1988-2003 encompasses much of the development of car seats from rickety metal frameworks that would look more at home next to the kitchen table to the modern formed plastic with shock absorbent foam models that we are familiar with today. Here are some examples of car seats in use during the sample period (sourced from here).

Techart – 1988

Fisher Price T-Shield – 1990

Renolux GT2000 – 1993
Evenflo Medallion – 2000

These car seats are all lumped together as if they were on the road at the same time. It is pretty apparent that both in general and in particular with respect to side impact crashes (a metric the report places quite a bit of emphasis on) a lot changed over the 15 year period sampled and also in the ensuing decade since. Do you think the first and last car seats have the same protective properties? It seems highly unlikely that they do even leaving aside the profound differences in tethering systems that prevailed between the decade and a half that the sample is taken across.

So, in short, this report provides no significant evidence about the efficacy of forward or rear facing child seat placement in terms of modern automobiles and child car seats. It MAY BE that there are some significant issues with respect to car seat direction, but the report that underpins the AAP recommendation that people are following is devoid of value as a piece of research.

And then finally, there is the issue of utility. Since my daughter has been facing forward in our car she is a much happier camper nearly all the time that we drive. We have little or no crying and just about any parent can tell you that a crying (or screaming) baby, particularly when you cannot see them at all makes for some very distracted driving. Most of the time now she just babbles to herself about the view. Better and safer for us both.

So, I concede that was perhaps a long blog post but, to paraphrase Chomsky, it takes time to contradict the conventional wisdom. The moral of this story I suppose is twofold. First, don’t be afraid to question putatively authoritative sources. Sometimes they are just plain wrong. We used to douse whole neighborhoods in DDT every summer to combat mosquitos, we used to give hard narcotics to babies with colic. These were not accidents, they were policies or recommended practices. Second, question any rules or policies that consider only risks and not benefits (or vice versa!). Very rarely planes crash and almost everyone always dies but no one is telling you that the safest policy is to not fly. Even more germane to this topic, if you want to be as safe as possible with your baby with respect to driving, simply abstain from any driving that is not absolutely essential to the health and well-being of your baby. That is the safest way, hands down.

Mostly, just think of life as it really is – full of trade-offs that we have to make every day – and make your decisions accordingly.

Thanks for reading!

* With the sole exception that we do most of our driving in Cook County, which has a disproportionate share of all Illinois accidents. Accidents per mile were unavailable to me on the county level so using statewide data may slightly understate our risk vis a vis driving in Cook County. But we also drive about half of our miles on the highway between Illinois and Wisconsin and, since highway driving is a good deal safer on average than urban driving, this abnormality is likely a wash.


About theunlikelyeconomist

theunlikelyeconomist is in the midst of the long slog to attain a PhD in economics.
This entry was posted in healthcare, statistics / data analysis and tagged , , , , . Bookmark the permalink.

4 Responses to How I learned to stop worrying and love the data

  1. brother of the author says:

    I can’t help but feel I may haveaided in setting this in motion. Can similar inaccuracies be said of the child/pool safety statistics argument?

    • Mmm, the pool thing is ambiguous since there isn’t data on fences, etc but looking here ( suggests that very brief lapses in parent/caregiver attention is the catalyst for toddler drowning deaths. I am really just of the opinions that water is deeply, deeply underpriced so that pools have negative externalities for society at large, particularly in water-constrained places like central Texas. Was probably just safety-baiting a bit when we discussed that based on my dislike of private pools. But I would be quite concerned about the minute to minute whereabouts of my child in a home with a below-ground pool.

  2. hyperspasm says:

    OK, so I haven’t even finished reading yet, but this popped out at me:

    1) Probabilty of being in a car crash: 1 in 2.6 million (1 / 2,600,000) per mile driven.
    We drive with our daughter an average of about 200 miles per month (a generous overestimate). This works out to 2400 miles per year. So our most basic probability of being in car crash with our daughter is 1 in 1083 annually. But there’s also this:
    2) Proportion of crashes between the hours of 8 AM and 8 PM: 72%
    This time period contains nearly 100% of the time we ever drive with our child in the car and serves to multiply the denominator of the above probability by 1.385. Thus our average probability of being in a car crash with any injury is closer to 1 in 1505 annually.

    Correct me if I’m wrong, but wouldn’t you want to multiply the denominator by 0.72 (or the numerator by 1.385)? The fact that you always drive between 8am and 8pm should INCREASE the chances of you having an accident in a given year from 1in1083 to 1in780. Right?

  3. I don’t think that is the case but I believe you have made me realize a different error. Since accidents are causally distributed across the day in the aggregate (daylight vs. dark, congestion related vs. drunk or tired driving related) not driving for a part of the day should reduce your chance of being involved in accidents correlated with a temporal condition. The problem I do have though is that I considered miles driven the same. So the original 1 in 2.6 million miles came from taking crashes and dividing them by total miles driven. If I want to exclude a portion of the day then I also needed to exclude the miles driven during that time of day. In this case miles driven by time of day is harder to find, but the National Household Travel Survey here ( finds that about 93% of driving is between 6am and 10pm. It is probably reasonable to assume from intuition that perhaps around 85% or so is between 8am and 8pm, so in that case what I need to do is the following:

    1/2,600,000 * 2,400 = 2,400/2,600,000. (2,400/2,400)/(2,600,000/2,400) = 1/1,083 (my original figure). Then I need to multiply 1/1,083 * .72/.85 (this is 72% of the accidents in 85% of the miles driven) = .72/921. (.72/.72)/(921/.72) = 1 in 1279.

    Thanks for making me think this through.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s