Post Hoc Ergo Propter Hoc

Bad Science and Public Policy

D. A. Clarke and Riley Geary
RTF version
Plot 1A, EPS
Plot 1B, EPS
Plot 2A, EPS
Plot 2B, EPS

Every student of logic (or science) is familiar with the famous fallacy called Coincidental Correlation, or post hoc ergo propter hoc. The literal translation of the Latin is "after this, therefore because of this," and it refers to the universal human tendency to ascribe causation where there is only sequence.

For example: if someone hangs the laundry out and then it rains, we may say jokingly that Murphy's Law was invoked by hanging out the laundry, thus causing the rain. This is one of a family of colloquial jests based on the fallacy of coincidental correlation. Human beings have a highly developed facility for pattern recognition; in fact, we often perceive pattern where there is none. [1] We also have a desire for predictability, and control over events. Often these attributes lead us to subscribe to the fallacy of coincidental correlation -- not in jest, but quite seriously.

Fortunately for the shamans, clairvoyants, astrologers and amateur weather forecasters of the world, people remember most clearly those events in which expectation or prediction was fulfilled, and tend to forget those events in which expectation was disappointed. If a would-be magus uses voodoo dolls to curse ten enemies, and one victim suddenly dies, this "success" will be remembered strongly and the nine failures discounted. Wishful thinking and overzealous pattern recognition both play a part in uncritical acceptance of post hoc ergo propter hoc.

Existing predisposition or prejudice also plays a strong part in post hoc credulity. If the observer is already biased towards a given conclusion, for example that "wet kites generate electricity," then a lightning strike on a kite flown during a thunderstorm will be seen as proving the preconceived theory.

It is not only in the realm of superstition, folklore, or bigotry that we encounter this fallacy. Scientists have often been fooled by apparent correlation where there was none. Government planners and forecasters are particularly susceptible to drawing optimistic causal conclusions from sequential data. An additional skew factor is introduced if authorities feel a need to justify, after the fact, strategies or interventions which were contentious and/or costly at the time.

Consider for example a group of politicians who supported e.g. a major and expensive highway widening project: they are likely to attribute to their own foresight and achievement any positive events which postdate the project completion. If the collision rate in the local area goes down after the new road is finished, it may very well be said in official reports and press releases that the new road has "improved safety". Their investment (professional, emotional, and financial) in the project creates an existing prejudice which will make post hoc reasoning attractive.

Rarely will such analyses factor in other contemporary events such as, for example, a police crackdown on drunk drivers, or a temporary rise in gasoline costs which caused people to drive less or drive more slowly. Only when two organizations are publicly contending for the credit for an improvement, is there a visible discussion of the relative likelihood of different causative factors.

Scientists are trained to treat post hoc results with extreme skepticism, considering every possible alternative explanation (and devising clever control cases) in an attempt to avoid false perception of cause where there is only sequence. The ideal, unbiased, highly-trained scientist should in theory be capable of applying this corrective to initial results and avoiding the publication of bad science. Most administrators and official functionaries, however, are neither highly trained in scientific methodology, nor unbiased. And even scientists can interpret sequential data optimistically, or selectively disregard part of a dataset; scientists are vulnerable to their own prejudices, despite their training.

The Johns Hopkins Medical Institute recently (Feb 20 2001) issued a press release announcing the completion of a paper on the effect of alcohol consumption on bicycle collisions. One of the co-authors, researcher Susan Baker, was quoted as follows[2]:

. . . while the number of fatal bicycle accidents in children has decreased by 70 percent since 1975, it has increased in adults by about 65 percent during the same time period.

The helmet laws that worked well for children should be extended to adult bicyclists, says Susan P. Baker, M.P.H., another author of the study . . .

Ms. Baker's argument is not unusual; on the contrary, it is the "received opinion" within the highway safety and law enforcment establishments in the US and has for many years been generally accepted as unassailable. The implicit reasoning is as follows: US child cycling fatalities have declined during the period since 1975; during the period since 1975, helmets were popularized and then mandated for juveniles in the US. Therefore, helmets were the reason for the reduction in child fatalities. The fact that adult cycling fatalities have not declined over the same period is therefore attributable to the exemption of adults from mandatory helmet laws; if adults had been legally obliged to wear helmets, the same "helmet effect" would have been seen in adult cycling fatalities. Therefore, the laws should be expanded to include adults.

Figures 1A and 1B

Actual US juvenile (under-16) traffic fatality statistics can be retrieved from various online sources, primarily the national Fatality Analysis Reporting System (FARS) database. (This database is maintained by the National Highway Traffic Safety Administration and is accessible to the public; see raw data and references below.) In Figures 1A and 1B above, we represent actual juvenile traffic fatality figures, from FARS and other sources, as a pair of time-series plots (see raw data below). We see that juvenile cyclist fatalities have indeed declined steadily from a high point in 1975. In contrast, juvenile motor vehicle fatality rates have varied randomly over the same period and do not show a clear declining trend.

So it does seem fairly obvious: as helmet use increased and helmet laws were enforced over this quarter-century, juvenile cycle fatalities declined. Therefore bike helmets are highly effective, and should be more heavily promoted and enforced.

There are two problems with this simple and obvious conclusion. One is that active bicycle helmet promotion campaigns did not commence in the US until the middle 1980's. A survey in 1991 found that only 18% of American cyclists wore helmets, but by 1998 a similar survey found that about 50% (38% of adults, 69% of juveniles under 16) of US cyclists were wearing helmets.[3] This narrow period of steep helmet take-up from about 1988 to 1998 is not reflected in the fairly constant slope of the decline in juvenile cyclist fatalities. This weakens even the classic post hoc fallacy, since there is not a clearly demarcated "before and after" pattern in the data.

If we take a less selective view of the dataset, a larger problem appears. If we include juvenile pedestrian fatalities as well, our time-series plots look like this:

Figures 2A and 2B

We now see that juvenile pedestrian fatalities actually declined by a slightly greater amount than juvenile bicycle fatalities, over the same 25-year period (72.4% vs 68.6%). Moreover, they declined at a rate that is remarkably similar to the decline in cyclist fatalities. From a purely statistical point of view, the correlation coefficient between the two trendlines is a remarkably positive .985 -- indeed, the two trendlines are so strongly correlated that it would be very surprising if they were not causally related to each other.

However, we know that there was no national program to put helmets on juvenile pedestrians. There was no publicity campaign in this direction, no advertising by helmet manufacturers directed at pedestrian use, and no laws enforcing helmet use by pedestrians. To what, then, can we ascribe the remarkably similar decline in pedestrian fatalities? And to what can we ascribe the constant decline in juvenile cycling fatalities during the years prior to any active pro-helmet campaign?

One thing we do know (from the CDC and other health-watching agencies) is that children in the US both walk and cycle far less than they used to. These agencies have repeatedly expressed concern in the last few years about a general decline in fitness among American children due to reduced physical activity. In 1999 health scientist Richard Killingsworth of the Center for Disease Control (USA) said "Biking as a form for children to get from one location to another has become very, very rare." He told the New York Times reporter Peter Kilborn [4] that

Less than 1 percent of children ages 7 to 15 ride bicycles to school . . . a precipitous and accelerating decline since the 1970's. Only 2.5 percent of those who live within two miles of school ride bikes there . . . For children . . . bicycles have been muscled aside by parental fears of crime and traffic, tight scheduling of organized play, television and computer games, and disappearing sidewalks.
The CDC has not altered its findings since then. "Nearly half of all young people do not take part in regular, vigorous physical activity," says the CDC in its current warnings about declining child physical health; this will be "a generation at risk for cardiovascular diseases, diabetes, and other serious health problems."[5] In February of 2001, Richard Killingsworth warned American physicians [6] that
Children have also suffered from dependence on the automobile, and while most children have bicycles, few use them for transportation. Over the last 20 years, nonmotorized trips made by children to school have declined by more than 40% (P. Schimek, unpublished calculations from the 1995 Nationwide Personal Transportation Survey, US Department of Transportation, Volpe Research Center, Cambridge, MA, 1999). This precipitous decline may hinder children's social, emotional, and physical development because it impedes their opportunity to engage in spontaneous outdoor physical activity. Furthermore, the absence of positive environmental cues for promoting physical activity may be a contributing factor in the burgeoning epidemic of overweight children. These children now must be chauffeured to places that traditionally were reached by foot or bicycle.

The proximate cause of this trend is easily identified: as sprawl has increased, American families have become more car-dependent and the number of miles driven per person per annum has increased significantly with each passing year. With more and more cars driving more and more miles, road systems have been re-engineered for more lanes and faster traffic. This in turn has created urban and suburban environments hostile to cycle and pedestrian transit: millions of Americans have been literally scared off the streets. Parents in particular have experienced increasing anxiety for their children's safety, and have restricted children's mobility more and more. Today, many parents in the US drive their children distances of a half-mile or less to school rather than permitting them to walk or cycle.

In this well-attested trend we find a simple explanation for the linked decline in juvenile walking and cycling fatalities. There are fewer kids on the streets walking and cycling, so fewer juvenile fatalities are experienced using those transit modes. It is not unreasonable to suppose that the restriction of child mobility has followed a smooth curve since 1975, along with the increase in motor traffic and distances driven per annum. This curve would then appear as it does on our plot -- unaffected by trends in helmet use.

In order to defend a "helmet effect" to which we attribute specifically the cycling fatality decline but not the pedestrian fatality decline, we would have to go through some statistical contortions. We would have to posit that the observed decline in the juvenile cycling fatality rate started diverging (for unknown reasons) from the nearly identical decline in the juvenile pedestrian fatality rate sometime in the mid/late 1980's -- just as helmets were starting to be marketed to young cyclists. We would then have to assume that the rate of divergence was almost exactly counterbalanced by the effectiveness of helmets, bringing the two trends back into their remarkable correlation.

Occam's Razor should dissuade us from seeking such a complicated and mysterious explanation when a simple and logical one is available. If we look only at the first, incomplete dataset, then we might be fooled by Coincidental Correlation. But to stick to that interpretation after considering the full dataset, requires a laboured and excessively complicated causation model.

If we resolutely refuse the simpler explanation and insist on this "mysterious divergence exactly cancelled by intervention" model, then there is probably some other motivation at work -- such as the wishful thinking or face-saving referred to above. People strongly wish to believe that simple, foolproof (purchasable) remedies exist for complicated social problems like traffic danger; so we are automatically inclined to feel that trivial engineering interventions like bicycle helmets should work. Authorities who have invested years and millions of dollars in "safety programs" promoting such interventions are strongly motivated to prove, after the fact, that they were right and that funds were not misspent nor time wasted.

The "safety" campaigns of the last 25 years have emphasized road danger; the traditional approach has been to frighten pedestrians and cyclists and instruct them to "keep out of the way of the cars," rather than attempting to modify the behaviour of drivers with a view to making the roads safer for more vulnerable users. Alternative approaches exist for cycling in particular: for example, there is the formalized "vehicular cycling" approach which emphasizes cyclist skill, compliance with vehicle code, and public awareness of cyclists' legal rights. Police and traffic authorities, however, have traditionally failed to emphasize in their "bicycle safety" campaigns even such basic issues as adequate night lighting; almost all publicly-funded effort has been invested in promoting public awareness of the "dangers" of cycling, and prescribing helmet use as the remedy.

It is ironically possible that this aggressive promotion of cycle helmets for children and adults -- which has redefined cycling in many people's minds as a highly dangerous activity -- did indeed contribute to the decline in juvenile bike and pedestrian fatalities. The helmet promotion effort may have had this effect not because helmets are mechanically effective in preventing fatalities, but because it reinforced and perpetuated the atmosphere of fear which the "safety" campaigns created. This "fear factor" has led to ever-increasing restrictions on children's freedom of movement, as mentioned above. [7]

To the student of classical logic, the use of juvenile road fatality data to "prove" that helmets are effective in reducing cycling fatalities is a perfect illustration of post hoc ergo propter hoc in daily life. We need not seek out superstitious tribespeople in deep jungles to confirm the enduring popularity of this fallacy. Nor is there any need to uncover far-reaching conspiracies concealing evidence for some nefarious end. Much of the bad science we too often encounter in the realm of public policy is attributable to simple human wishfulness -- plus the timeless appeal of the Coincidental Correlation.


NOTES

  1. Hake H.W. and Hyman R. (1953), "Perception of the statistical structure of a random series of binary symbols", Journal of Experimental Psychology, 45, 64-74.
  2. Science Daily February 2001.
  3. CPSC national helmet use surveys While there are serious methodological problems with these surveys, they are at least comparable to one another, so the overall trend is clear even if the actual numbers are dubious.
  4. New York Times June 7, 1999: "No Work for a Bicycle Thief" by Peter T. Kilborn
  5. See CDC "Walk Our Children to School" program
  6. See Community Design and Transportation Policies: New Ways to Promote Physical Activity in The Physician and Sportsmedicine Feb 2001 (guest editorial)
  7. See "The Impact of Transport Policy on Children's Development, Mayer Hillman 1999; also Hillman, Adams, and Whitelegg, One False Move: A Study of Children's Independent Mobility, Policy Studies Institute 1990, and the compendium "Children, Transport and the Quality of Life", proceedings of PSI conference 1993, various authors.

RAW DATA FROM FARS AND IIHS

Extracted by Riley Geary (Institute for Traffic Safety Analysis) from the FARS online database (1994-1999) and the Insurance Institute for Highway Safety (1975-1993):

Yr      J-MV    J-Ped   J-Bike  P/B     J-M%    J-P%    J-B%

75      2256    2056    682     3.02    100.0%  100.0%  100.0%
76      2189    1918    612     3.13    97.0%   93.3%   89.7%
77      2290    1751    618     2.83    101.5%  85.2%   90.6%
78      2274    1788    571     3.13    100.8%  86.9%   83.7%
79      2170    1627    559     2.91    96.2%   79.1%   82.0%
80      2170    1491    531     2.81    96.2%   72.5%   77.9%
81      1938    1341    496     2.70    85.9%   65.2%   72.7%
82      1888    1242    415     2.99    83.7%   60.4%   60.9%
83      1848    1189    440     2.70    81.9%   57.8%   64.5%
84      1762    1178    419     2.81    78.1%   57.3%   61.4%
85      1844    1166    435     2.68    81.7%   56.7%   63.8%
86      1936    1150    446     2.58    85.8%   55.9%   65.4%
87      2139    1131    442     2.56    94.8%   55.0%   64.8%
88      2207    1140    396     2.88    97.8%   55.5%   58.1%
89      2394    990     370     2.68    106.1%  48.2%   54.3%
90      2067    968     299     3.24    91.6%   47.1%   43.8%
91      2011    876     307     2.85    89.2%   42.6%   45.0%
92      2010    800     302     2.65    89.1%   38.9%   44.3%
93      1990    818     310     2.64    88.2%   39.8%   45.5%
94      2256    806     299     2.70    100.0%  39.2%   43.8%
95      2257    754     280     2.69    100.0%  36.7%   41.1%
96      2266    715     248     2.88    100.4%  34.8%   36.4%
97      2222    644     250     2.58    98.5%   31.3%   36.7%
98      2152    580     230     2.52    95.4%   28.2%   33.7%
99      2001    567     214     2.65    88.7%   27.6%   31.4%
The columns are as follows: Year; Juvenile motor vehicle occupant fatalities; Juvenile pedestrian fatalities; Juvenile cyclist fatalities; ratio of pedestrian to cycle fatalities. The last three columns show the percentage of Year 1 (1975) fatalities represented by each subsequent year of Motor Vehicle, Pedestrian, and Bike Juvenile Fatalities, or in other words, what is the amount and direction of change since 1975. The paired plots above show columns 2, 3, and 4 (top plot) and columns 6, 7, and 8 (bottom plot).

We should perhaps note in passing that there was a demographic shift (Baby Boomers to Gen X-ers) which reduced the size of the US population 14 years of age and younger by about 9% over the period 1970-1990. We should expect this reduction in the size of the juvenile demographic to be reflected equally in all trends on our graph; but it hardly accounts for declines on the order of 70 percent in juvenile bike and pedestrian fatalities.

This phenomenon is not limited to the United States. It is interesting to plot the same type of data (juvenile road fatality trends) from another motorized nation, such as the United Kingdom.

Figure 3

Here we see a rather similar gradual decline in juvenile pedestrian and cyclist fatalities. In the UK, child motor vehicle occupant fatalities are also slowly declining, whereas in the US they are holding steady. In the UK, there has never been a bike helmet law, neither for adults nor for children; but in the UK, private automobile use has steadily been displacing independent juvenile mobility just as it has in the US, with the same result: a decline in exposure leading to a decline in fatalities. In the UK there is a similar expression of concern by medical professionals over potentially grave long-term health implications of the sedentary automobile-centered lifestyles of both adults and children.


de@daclarke.org
De Clarke