by Nate Silver
Meanwhile we theorized that, if Japan seemed to be mobilizing for an attack, it would be against Russia or perhaps against the Asian territorial possessions of the United Kingdom, Russia and the UK being countries that were already involved in the war. Why would the Japanese want to provoke the sleeping giant of the United States? We did not see that Japan believed our involvement in the war was inevitable,14 and they wanted to strike us when we were least prepared and they could cause the most damage to our Navy. The imperial Japanese government of the time was not willing to abandon its hopes for territorial expansion. We had not seen the conflict through the enemy’s eyes.
To Wohlstetter, a signal is a piece of evidence that tells us something useful about our enemy’s intentions;15 this book thinks of a signal as an indication of the underlying truth behind a statistical or predictive problem.* Wohlstetter’s definition of noise is subtly different too. Whereas I tend to use noise to mean random patterns that might easily be mistaken for signals, Wohlstetter uses it to mean the sound produced by competing signals.16 In the field of intelligence analysis, the absence of signals can signify something important (the absence of radio transmissions from Japan’s carrier fleet signaled their move toward Hawaii) and the presence of too many signals can make it exceptionally challenging to discern meaning. They may drown one another out in an ear-splitting cacophony.
The next set of diagrams consist of a series of ten signals, each governed by an extremely simple and orderly mathematical function known as a sine wave. In figure 13-2a, one of these signals is highlighted and is much more distinct than the others. After the fact of an attack or some other failure of prediction, this is how the world may look to us. We will see the signal: the paper trail, the pattern, the precursors. Following both Pearl Harbor and September 11, a significant minority of Americans asserted that the patterns were so clear that the government must have known about the attack, and therefore must have been complicit in planning or executing it.*
FIGURE 13-2A: COMPETING SIGNALS WITH ONE SIGNAL HIGHLIGHTED
FIGURE 13-2B: COMPETING SIGNALS, UNDIFFERENTIATED
But this is not usually how the patterns look to us in advance. Instead, they are more like figure 13-2b, an ugly mess of tangled spaghetti string. As Wohlstetter writes:17
It is much easier after the event to sort the relevant from the irrelevant signals. After the event, of course, a signal is always crystal clear; we can now see what disaster it was signaling, since the disaster has occurred. But before the event it is obscure and pregnant with conflicting meanings. It comes to the observer embedded in an atmosphere of “noise,” i.e., in the company of all sorts of information that is useless and irrelevant for predicting the particular disaster.
In cases like these, what matters is not our signal detection capabilities: provided that we have met some basic threshold of competence, we will have perceived plenty of signals before something on the scale of Pearl Harbor or September 11. The relevant signals will be somewhere in a file cabinet or a computer database. But so will a whole host of irrelevant ones. We need signal analysis capabilities to isolate the pertinent signals from the echo chamber.
Usually, we will have some views on which signals are more important and require our focus. It is good and necessary to have these views, up to a point. I’ve detailed the problems that ensue when we consider data without context. Rather than make useful predictions, we trip out on the patterns and get nowhere fast.
However, the context we provide can be biased and self-serving. As Cicero warned Shakespeare’s Julius Caesar,18 “Men may construe things, after their fashion / Clean from the purpose of the things themselves.” We may focus on those signals which advance our preferred theory about the world, or might imply a more optimistic outcome. Or we may simply focus on the ones that fit with bureaucratic protocol, like the doctrine that sabotage rather than an air attack was the more likely threat to Pearl Harbor.
The Unfamiliar and the Improbable
Rumsfeld’s favorite part of Wohlstetter’s book is the foreword, composed by the Nobel Prize–winning economist Thomas Schelling, who was instrumental in translating John Nash’s early work on game theory into national-security contexts. Schelling writes of our propensity to mistake the unfamiliar for the improbable:
There is a tendency in our planning to confuse the unfamiliar with the improbable. The contingency we have not considered seriously looks strange; what looks strange is thought improbable; what is improbable need not be considered seriously.
Because of the United States’ isolation from the European and Asian continents and the relatively good relations we have maintained with the rest of the Americas since the promulgation of the Monroe Doctrine, we have infrequently been the subject of foreign attack. The exceptions (September 11) and near-misses (the Cuban Missile Crisis) have therefore been exceptionally jarring to us. Before Pearl Harbor, the last foreign attack on American soil had been during the War of 1812.19 Americans just do not live among the ruins of wars past, as people in Europe and Asia have throughout their history.
But our Hawaiian territory* sat in the middle of the Pacific Ocean: Honolulu is closer to Tokyo (3,860 miles) than to Washington, DC (4,825 miles). Because of its geographic position and the presence of our naval fleet, it made an obvious target to the Japanese. The unfamiliarity of attacks on American territory may have made us complacent about the threat.
Perhaps we went through a logical deduction something along these lines:
The United States is rarely attacked
Hawaii is a part of the United States
Therefore, Hawaii is unlikely to be attacked
This is deeply flawed thinking. As I described in chapter 1, our predictions often fail when we venture “out of sample.” The fact that the United States had rarely been attacked is an empirical observation, not an iron law. That Nebraska, say, had never been attacked by a foreign power gave no real evidentiary weight to the situation in Hawaii, given the latter’s outlying geographic position in the Pacific and the precariousness of the war situation there.
But at least this flawed type of thinking would have involved some thinking. If we had gone through the thought process, perhaps we could have recognized how loose our assumptions were. Schelling suggests that our problems instead run deeper. When a possibility is unfamiliar to us, we do not even think about it. Instead we develop a sort of mind-blindness to it. In medicine this is called anosognosia:20 part of the physiology of the condition prevents a patient from recognizing that they have the condition. Some Alzheimer’s patients present in this way.
The predictive version of this syndrome requires us to do one of the things that goes most against our nature: admit to what we do not know.
Was 9/11 a Known Unknown?
[T]here are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—there are things we do not know we don’t know.—Donald Rumsfeld21
Rumsfeld’s famous line about “unknown unknowns,” delivered in a 2002 press conference in response to a reporter’s question about the presence of weapons of mass destruction in Iraq, is a corollary to Schelling’s concern about mistaking the unfamiliar for the unlikely. If we ask ourselves a question and can come up with an exact answer, that is a known known. If we ask ourselves a question and can’t come up with a very precise answer, that is a known unknown. An unknown unknown is when we haven’t really thought to ask the question in the first place. “They are gaps in our knowledge, but gaps that we don’t know exist,” Rumsfeld writes in his 2011 memoir.22
The concept of the unknown unknown is sometimes misunderstood. It’s common to see the term employed in formulations like this, to refer to a fairly specific (but hard-to-predict) threat:
Nigeria is a good bet for a crisis in the not-too-distant future—an unknown unknown that poses the most profound implications for US and global security [emphasis add
ed].23
This particular prophecy about the terrorist threat posed by Nigeria was rather prescient (it was written in 2006, three years before the Nigerian national Umar Farouk Abdulmutallab tried to detonate explosives hidden in his underwear while aboard a flight from Amsterdam to Detroit). However, it got the semantics wrong. Anytime you are able to enumerate a dangerous or unpredictable element, you are expressing a known unknown. To articulate what you don’t know is a mark of progress.
Few things, as we have found, fall squarely into the binary categories of the predictable and the unpredictable. Even if you don’t know to predict something with 100 percent certainty, you may be able to come up with an estimate or a forecast of the threat. It may be a sharp estimate or a crude one, an accurate forecast or an inaccurate one, a smart one or a dumb one.* But at least you are alert to the problem and you can usually get somewhere: we don’t know exactly how much of a terrorist threat Nigeria may pose to us, for instance, but it is probably a bigger threat than Luxembourg.
The problem comes when, out of frustration that our knowledge of the world is imperfect, we fail to make a forecast at all. An unknown unknown is a contingency that we have not even considered. We have some kind of mental block against it, or our experience is inadequate to imagine it; it’s as though it doesn’t even exist.
This poses especially grave risks when we consider the signals we receive from terrorists. As before Pearl Harbor, there were many signals that pointed toward the September 11 attacks:
There had been at least a dozen warnings24 about the potential for aircraft to be used as weapons, including a 1994 threat by Algerian terrorists to crash a hijacked jet into the Eiffel Tower, and a 1998 plot by a group linked to Al Qaeda to crash an explosives-laden airplane into the World Trade Center.
The World Trade Center had been targeted by terrorists before. The 1993 bombings by Ramzi Yousef and his co-conspirators, who had trained at Al Qaeda camps in Afghanistan, had killed six and were intended to bring the Twin Towers down.25
Al Qaeda was known to be an exceptionally dangerous and inventive terrorist organization. It had a propensity for pulling off large-scale attacks, including the bombing of U.S. embassies in Kenya and Tanzania in 1998, which killed 224, and the bombing of the USS Cole in Yemen in 2000.26
Secretary of State Condoleezza Rice had been warned in July 2001 about heightened Al Qaeda activity—and that the group was shifting its focus from foreign targets to the United States itself.27 “It’s my sixth sense,” CIA chief George Tenet had said after seeing the intelligence. “But I feel it coming. This is going to be the big one.”28
An Islamic fundamentalist named Zacarias Moussaoui had been arrested on August 16, 2001, less than a month before the attacks, after an instructor at a flight training school in Minnesota reported he was behaving suspiciously.29 Moussaoui, despite having barely more than 50 hours of training and having never flown solo, had sought training in a Boeing 747 simulator, an unusual request for someone who was nowhere near obtaining his pilot’s license.30
It is much easier to identify the importance of these signals after the fact; our national security agencies have to sort through literally tens of thousands or even hundreds of thousands of potential warnings31 to find useful nuggets of information. Most of them amount to nothing.
Still, the September 11 plot was exceptionally ambitious—and the terrorists were able to execute it with relatively few hitches. Nineteen terrorists had entered the air-transit system, successfully hijacking four planes. Three of the four planes hit their targets; United 93 failed to do so only because of the exceptional bravery of the passengers on board, who charged the cockpit after they learned what had happened to the other flights.* Not only had we failed to detect the plot, but it doesn’t appear that we came all that close to doing so.
The 9/11 Commission Report identified four types of systemic failures that contributed to our inability to appreciate the importance of these signals, including failures of policy, capabilities, and management.32 The most important category was failures of imagination.33 The signals just weren’t consistent with our familiar hypotheses about how terrorists behaved, and they went in one ear and out the other without our really registering them.
The North American Aerospace Defense Command (NORAD) had actually proposed running a war game in which a hijacked airliner crashed into the Pentagon. But the idea was dismissed as being “too unrealistic.”34 And in the unlikely event that such an attack were to occur, it was assumed, the plane would come from overseas and not from one of our domestic airports. (Ironically, this was the exact opposite of the mistake that we’d made before Pearl Harbor, where the possibility of an attack from abroad was dismissed because planners were concerned about sabotage.)
The possibility of a suicide attack may also have been hard to imagine. FAA policy was predicated on the idea that a hijacking would result in a tense standoff and perhaps a detour to some exotic airport in the Middle East. But it was assumed the terrorist would not want to destroy the plane, or to kill passengers other than as a negotiation tactic. Thus, cockpit doors were not tightly sealed and were often left entirely unlocked in practice.35
Yet suicide attacks had a rich history36—including, of course, the Japanese kamikaze pilots in World War II.37 Moreover, suicide attacks had become much more common in the years immediately preceding September 11; one database of terrorist incidents38 documented thirty-nine of them in 2000 alone, including the bombing of the USS Cole in Yemen, up from thirty-one in the 1980s.
However, World War II was a distant memory, and most of the suicide attacks had occurred in the Middle East or in Third World countries. The mental shortcut that Daniel Kahneman calls the availability heuristic39—we tend to overrate the likelihood of events that are nearer to us in time and space and underpredict the ones that aren’t—may have clouded our judgment.
“You can reasonably predict behavior if people would prefer not to die,” Rumsfeld told me. “But if people are just as happy dying, or feel that it’s a privilege or that it achieves their goal, then they’re going to behave in a very different way.”
The Scale of September 11
The September 11 plot, then, was less a hypothesis that we evaluated and rejected as being unlikely—and more one that we had failed to really consider in the first place. It was too unfamiliar to us. Rumsfeld suggested in his memoir that September 11 was an unknown unknown.40
“The CIA would tell you that it was not a thought that was completely foreign to them,” he told me. “But a lot of it came after the attack, in retrospect.”
“I mean, it’s fair to say Al Qaeda was a known unknown,” added Urbahn, Rumsfeld’s chief of staff. “But one of the things [Tenet] talks about in his book is that the scale was far different than any other attack. The fact that it was—so big.”
Indeed, the sheer magnitude of the September 11 attack—2,977 innocent people killed—most differentiated it from our previous experience with terrorism. Even those who were the most concerned about what Al Qaeda was planning, like Tenet and Richard Clarke, the chairman of counterterrorism at the National Security Council, had trouble conceiving of its scale. The notes that Clarke had sent to Condoleezza Rice, for instance, implored her to imagine what might happen if hundreds of Americans died41—not thousands of Americans, as actually occurred.
Prior to September 11, the largest terror attack in a Western country* had been only about one-tenth as fatal, when in 1985 a Sikh extremist group hid a bomb in an Air India flight bound for Delhi from Montreal, killing 329 passengers. The destruction of Oklahoma City’s Alfred P. Murrah federal building by the militant Timothy McVeigh in 1995, which killed 168, had been the largest attack on American soil.
But September 11 was not an outlier. Although the particulars of the events that day were not discerned in advance—and although they may have been very hard to predict—we had some reason to think that an attack on the scale of September 11 was possible.
The Mathema
tics of Terrorism: Why 9/11 Wasn’t an Outlier
It might seem uncomfortable to think about terrorism in an abstract and mathematical fashion, as we are about to do here. To be clear, this is not a substitute for the sort of signal analysis that the intelligence community performs. But this type of thinking can help patch some of our blind spots and give us better estimates of the overall hazard that terrorism poses. We can better appreciate future risk if we analyze the existing data.
In 2008, I was invited to speak at a conference hosted by the Center for Strategic and International Studies (CSIS), a Washington-based foreign policy think tank. The timing couldn’t have been much worse—it was just two weeks before the 2008 elections—but I was told that the panel involved national security and I suppose I felt it was my duty to attend.
The conference gathered experts from different domains in the hope that a group brainstorming session might stumble across one or two “outside-the-box” insights about how to predict and prevent terror attacks. There was a marketing executive from Coca-Cola on the panel, a police detective from New York, a man who created algorithms for the dating Web site eHarmony—and me. (There were also a number of experts whose work was more self-evidently related to terrorism: people who worked for the State Department, or the military, or for defense contractors around Washington, DC.)
I gave a short presentation describing my work predicting outcomes in baseball and in politics; it was received politely enough. But then it was time for the question-and-answer session. “That’s all very nice, Nate,” I was told. “But how in the hell is this applicable to terrorism?” (I’m paraphrasing, but only slightly.)