A young statistician saved their lives.
His insight (and how it can change yours):
During World War II, the U.S. wanted to add reinforcement armor to specific areas of its planes.
Analysts examined returning bombers, plotted the bullet holes and damage on them (as in the image below), and came to the conclusion that adding armor to the tail, body, and wings would improve their odds of survival.
But a young statistician named Abraham Wald noted that this would be a tragic mistake. By only plotting data on the planes that returned, they were systematically omitting the data on a critical, informative subset: The planes that were damaged and unable to return.
• The “seen” planes had sustained damage that was survivable.
• The “unseen” planes had sustained damage that was not.
Wald concluded that armor should be added to the *unharmed* regions of the returning planes (the areas without bullet holes on the image below).
His profound logic: Where the survivors were unharmed was actually where the planes were most vulnerable.
Based on his insight, the military reinforced the engine and other vulnerable parts, significantly improving the safety of the crews during combat and saving thousands of lives.
Abraham Wald had identified a cognitive bias called “Survivorship Bias“: The error resulting from systematically focusing on survivors (successes) and ignoring casualties (failures) that causes us to miss the true base rates of survival (the actual probability of success) and arrive at flawed conclusions.
We see examples of Survivorship Bias all around us:
- We read books on the common traits of successful people, but fail to consider all of the unsuccessful people who possessed those same traits.
- We applaud the belief when we hear that an entrepreneur took out a second mortgage and succeeded, but fail to consider all of the entrepreneurs who did the same and went bankrupt.
- We study the cultural strategies of the most successful companies, but fail to consider all of the companies that followed those same strategies and fell apart.
When we fail to consider the range of outcomes and the hidden evidence, we develop a skewed (and often incorrect) view of reality.
It cannot be avoided altogether, because the vast majority of books and history are written by and about the survivors and victors, but wherever possible, consider the unseen evidence. What is unseen often has just as much value as what is seen.
Credits:
https://en.wikipedia.org/wiki/Survivorship_bias
Wald, Abraham. Archived 2019-07-13 at the Wayback Machine. Center for Naval Analyses.
Wallis, W. Allen (1980). “The Statistical Research Group, 1942-1945: Rejoinder”. Journal of the American Statistical Association.
“Bullet Holes & Bias: The Story of Abraham Wald”. mcdreeamie-musing
“AMS :: Feature Column :: The Legend of Abraham Wald”. American Mathematical Society
‘How Not to Be Wrong’ by Jordan Ellenberg (released in 2014)
