Beware Survivorship Bias
Suppose you wanted to know how many of the soldiers, sailors, and airmen who served during World War II were killed in that war. So you sent inquiries to a random sample of persons whose names were drawn from a list of all those who served in the military during the war, asking: Were you killed in the war? I presume that all of those who responded to the survey would reply, no. Having conducted your scientific poll, you could then conclude that none of the soldiers who served in the military services during World War II were killed.
The mistake you would have made in this case springs from what is known as survivorship bias. It affects many sorts of studies, including many where the study design is not so obviously stupid as in my foregoing example. Surveys have sought, for example, to determine how an increase in the legal minimum wage affected employers’ total hours of labor services hired. Such a forced wage-rate increase, however, especially if it were a large one, might well cause some marginal firms to go out of business. They would then be unavailable to respond to a poll or to show up in another type of survey to indicate that the increased minimum wage had caused them to reduce their hours of employment to zero, wiping out however many jobs they had previously supported.
You might think that any well-trained economist would be aware of survivorship bias and would not draw unwarranted conclusions by failing to take it into account in designing, conducting, and interpreting a study. But if you thought so, you’d be wrong. Mainstream economists, including super-duper econometricians, not uncommonly make this freshman mistake.
Long ago, Frederic Bastiat famously warned against ignoring the unseen effects and focusing exclusively on the visible effects of government’s or others’ actions in markets. His warning is often ignored, however, even by professional economists, many of whom pride themselves on their exclusive focus on quantitative data—for them, if it can’t be (and hasn’t been) counted, it does not exist. Such an approach to evidence and economic reasoning is indefensible.
Much of what we economists know can be known directly from praxeology, the pure logic of choice most notably developed by Ludwig von Mises and his followers. Thus, if someone fails to see and measure, for example, employment losses in the wake of a substantial increase in the legal minimum wage, the sound economist’s reaction to this (non)observational report is not to suppose bizarrely that the law of demand does not apply to labor services, but to challenge the obsession with observed and counted employment reductions. Many of the effects of increasing the legal minimum wage, for example, take the form of actions that never occurred and hence cannot be observed, for example, jobs that were never created because at the higher minimum wage entrepreneurs did not consider the formation of certain types of new firms or the creation of certain types of new jobs to be worthwhile.
In short, in gaining a solid understanding of economic events, we must beware of survivorship bias and never fail to consider the unseen as well as the seen consequences of government interventions in the market. A corollary is that we must not fool ourselves into the naïve positivist belief that only countable data deserve consideration in scientific work. The seen and the unseen, the counted and the uncounted—all are proper raw materials for the serious and properly trained student of economic and social life.
