The Signal and the Noise

by Nate Silver

  Frankly, the methods I presented at the conference probably weren’t all that useful for national security analysis. Baseball and politics are data-rich fields that yield satisfying answers. Thousands of baseball games are played every year. Elections occur less frequently—and require somewhat more caution to forecast—but there are hundreds of polls released during every presidential campaign. All this data is publicly available, freely or cheaply.

  Terrorism, seemingly, is not this way at all. The events that we are really concerned about, like September 11, occur only rarely. Moreover, terrorist groups try to conceal their intentions—Al Qaeda was known to be especially effective at this. Within the realm of terrorism, just as was the case immediately before Pearl Harbor, the absence of signals is sometimes more worrisome than the presence of them. If the CIA is able to hack into an Internet chat site thought to be used by radical groups, there may be quite a bit of loose and casual chatter at first, when organizations like Al Qaeda are simply hoping to find new and naïve recruits. But when an attack is being plotted, and the stakes are raised, the conversation usually goes offline.

  At a microscopic level, then—at the level of individual terrorists, or individual terror schemes—there are unlikely to be any magic-bullet solutions to predicting attacks. Instead, intelligence requires sorting through the spaghetti strands of signals that I spoke about earlier. One expert I talked to at the CSIS conference provided another metaphor: detecting a terror plot is much more difficult than finding a needle in a haystack, he said, and more analogous to finding one particular needle in a pile full of needle parts.

  Some problems that are highly unpredictable on a case-by-case basis may seem more orderly if we zoom out and take a macroscopic view. Outside insight into the mathematical properties of terrorism has proved more useful here.

  Aaron Clauset, a professor at the University of Colorado with a background in physics and computer science, has published papers on the mathematics of everything from the evolution of whales42 to the network dynamics of multiplayer role-playing games.43 The intelligence community has a long-standing reputation for preferring alpha-male types. The thirty-something Clauset, whose catholic research interests might seem geeky to it, has met a mixture of encouragement and resistance when presenting his findings.

  “Some people say it’s a breath of fresh air,” Clauset told me when we caught up on the phone. “That’s a few people. Most people look at it and say, ‘That’s kind of weird. You want to use math?’”

  Clauset’s insight, however, is actually quite simple—or at least it seems that way with the benefit of hindsight. What his work found is that the mathematics of terrorism resemble those of another domain discussed in this book: earthquakes.

  Imagine that you live in a seismically active area like California. Over a period of a couple of decades, you experience magnitude 4 earthquakes on a regular basis, magnitude 5 earthquakes perhaps a few times a year, and a handful of magnitude 6s. If you have a house that can withstand a magnitude 6 earthquake but not a magnitude 7, would it be right to conclude that you have nothing to worry about?

  Of course not. According to the power-law distribution that these earthquakes obey, those magnitude 5s and magnitude 6s would have been a sign that larger earthquakes were possible—inevitable, in fact, given enough time. The big one is coming, eventually. You ought to have been prepared.

  Terror attacks behave in something of the same way. The Lockerbie bombing and Oklahoma City were the equivalent of magnitude 7 earthquakes. While destructive enough on their own, they also implied the potential for something much worse—something like the September 11 attacks, which might be thought of as a magnitude 8. It was not an outlier but instead part of the broader mathematical pattern.

  Defining and Measuring Terrorism

  To look at the statistics of terrorism, we first need to define exactly what it is. This can get a little tricky. Vladimir Lenin said that “the purpose of terrorism is to terrorize.”44 This is more insightful than it might seem: terrorists are not purely seeking to maximize their body count; instead, they want to maximize the amount of fear they inflict on a population so as to alter its behavior in some way. Death and destruction are just a means toward that end. “You may kill people to achieve that,” Rumsfeld told me. “But that is not its purpose.”

  Still, there is a wide variety of violent behavior throughout the world, and so academics have sought a somewhat more precise definition to distinguish terrorism from its counterparts. One definition, employed by a widely used database of terrorist incidents,45 requires that terrorist acts must be intentional, that they must entail actual or threatened violence, and that they must be carried out by “subnational actors” (meaning, not directly by sovereign governments themselves). The incidents, moreover, must be aimed at attaining a political, economic, social, or religious goal. And they must involve some element of intimidation or coercion—intended to induce fear in an audience beyond the immediate victims.

  The type of terrorism that most explicitly meets these criteria and which is most familiar to us today has relatively modern roots. The UCLA political scientist David C. Rapoport dates it to 197946—the year of the Iranian Revolution. He relates it to religious extremism, particularly among Islamist groups. This wave of terrorism is associated with a sharp rise in the number of attacks against Western countries and Western interests; from 1979 through 2000 the number of terror attacks against NATO countries rose almost threefold.

  Most of the incidents, however, produced few fatalities, if any. From the Iranian Revolution through September 10, 2001, there were more than 4,000 attempted or successful terror attacks in NATO countries. But more than half the death toll had been caused by just seven of them. The three largest attacks—the Air India disaster, the Lockerbie bombing, and Oklahoma City—had accounted for more than 40 percent of the fatalities all by themselves.

  This type of pattern—a very small number of cases causing a very large proportion of the total impact—is characteristic of a power-law distribution, the type of distribution that earthquakes obey. Clauset’s insight was that terror attacks abide by a power-law distribution as well.

  Suppose that we draw a graph (figure 13-4) plotting the frequency of terror attacks on one axis and their death tolls on the other. At first, this doesn’t seem terribly useful. You can clearly see the power-law relationship: the number of attacks decreases very steeply with their frequency. But the slope is so steep that it seems to obscure any meaningful signal: you see a large number of very small attacks, and a small number of very large ones, with seemingly little room in between. The September 11 attacks look like an outlier.

  However, as was the case for earthquakes, the data is easier to comprehend when we plot it on a logarithmic scale (more specifically, as in figure 13-5, a double-logarithmic scale in which both the vertical and the horizontal axes are logarithmic). It’s important to emphasize that I’ve done nothing to this data other than make it easier to visualize—it’s still the same underlying information. But what had once seemed chaotic and random is now revealed to be rather orderly. When plotted on a double-logarithmic scale, the relationship between the frequency and the severity of terror attacks appears to be, more or less,47 a straight line. This is, in fact, a fundamental characteristic of power-law relationships: when you plot them on a double-logarithmic scale, the pattern that emerges is as straight as an arrow.

  Power laws have some important properties when it comes to making predictions about the scale of future risks. In particular, they imply that disasters much worse than what society has experienced in the recent past are entirely possible, if infrequent. For instance, the terrorism power law predicts that a NATO country (not necessarily the United States) would experience a terror attack killing at least one hundred people about six times over the thirty-one-year period from 1979 through 2009. (This is close to the actual figure: there were actually seven such attacks during this period.) Likewise, it implies that an attack that
killed 1,000 people would occur about once every twenty-two years. And it suggests that something on the scale of September 11,48 which killed almost 3,000 people, would occur about once every forty years.

  It’s not that much of an accomplishment, however, to describe history in statistical terms. Sure, it’s possible for a statistical model to accommodate an event like September 11 now that one has actually occurred. But what would Clauset’s method have said about the possibility of such an attack before it happened?

  September 11 certainly did shift the probabilities somewhat—just as the number of very large earthquakes in recent years implies that they are somewhat more common than we might have thought previously.49 Nevertheless, even before it occurred, the power-law method would have concluded that an attack on the scale of September 11 was a clear possibility. If the power-law process is applied to data collected entirely before 9/11—everything from the beginning of the modern wave of terrorism in 1979 through September 10, 2001—it implies that a September 11–scale attack would occur about once every eighty years in a NATO country, or roughly once in our lifetimes.50

  This method does not tell us anything specific about exactly where and when the attack would occur. It is a long-term tendency, like the tendency toward earthquakes in California. And terror attacks, unlike earthquakes, can plausibly be prevented—this is an important qualification to Clauset’s hypothesis.

  What this data does suggest is that an attack on the scale of September 11 should not have been unimaginable. The power-law distribution demonstrates that events of a much larger scale than occurred in the past may plausibly occur in the future. Our lack of familiarity with them will be an especially poor guide to their likelihood.

  Magnitude 9 Terrorism

  But if the September 11 attacks were tantamount to a magnitude 8 earthquake, what about the potential for something larger still: the equivalent of a magnitude 9? Clauset’s method gives us reason to believe that attacks that might kill tens of thousands or hundreds of thousands of people are a possibility to contemplate as well. The mechanism for such an attack is unpleasant but easy enough to identify. It would most likely involve weapons of mass destruction, particularly nuclear weapons.

  The world, fortunately, hasn’t had much experience with nuclear warfare. The atomic bombs dropped on Hiroshima and Nagasaki, Japan, in 1945 at the end of World War II killed about 200,000 people.51 One estimate52 holds that an equally powerful weapon, detonated at one of New York City’s major ports, would kill on the order of 250,000 civilians. But nuclear technology has evolved since then. A larger and more modern weapon, blown up over midtown Manhattan, could kill as many as one million New Yorkers53 while incinerating hundreds of billions of dollars in real estate. Simultaneous attacks on New York, Washington, Chicago, and Los Angeles could kill on the order of four million Americans, in line with a stated goal of Osama bin Laden.54

  These estimates reflect—probably—worst-case scenarios. Because such an attack could be hundreds of times worse than 9/11, however, the question of its likelihood has been the subject of intense debate in the national security community.

  One of the more pessimistic assessments comes from Graham Allison, a professor of political science at Harvard. Allison served in the administrations of both President Reagan and President Clinton, and his books and papers on the Cuban missile crisis have been cited thousands of times by other scholars.55 So when Allison has something to say, his peers listen.

  Allison came to a frightening conclusion in 2004: “A nuclear terrorist attack on America in the decade ahead is more likely than not,”56 he wrote. Allison qualified his forecast by saying it assumed we remained “on the current path”—a world in which there are dangerous terrorist groups, vulnerable nuclear materials at many places around the world, and a lack of focus on the problem from U.S. policy makers.

  We are more than a decade removed from the September 11 attacks. But when I spoke with Allison in 2010, he reaffirmed the gravity of threat he perceives. It’s one that he takes quite literally, in fact. I called Allison from my desk at the New York Times office, a block from Times Square.57 Allison told me that he’d be at least a little bit nervous in Times Square and wasn’t sure if he’d be willing to work there every day.

  Allison’s probability estimate doesn’t come from a statistical model. Instead it’s “the basis on which [he]’d make bets.”* Why does he see so much risk? “It’s a knockoff of the old Sherlock Holmes version of motive, means, and opportunity,” he told me.

  The motive of terrorists, for Allison, is easy to discern. Osama bin Laden had said that he wanted to kill four million Americans, a number that could probably be achieved only through a nuclear attack. The modus operandi of Al Qaeda has been what Allison calls “spectaculars”—occasional but astonishing attacks that kill large numbers of innocent people. And the CIA had picked up Al Qaeda chatter about an “American Hiroshima” before the September 11 attacks.58

  By opportunity, Allison means the ability of terrorist groups to smuggle a weapon into the United States. He has little doubt that this could happen. “How do crooks get into American cities every day?” Allison asked me. The United States has more than 3,700 ports, and receives more than six million cargo containers per year—but only 2 percent of them are physically inspected by customs agents.59 “If you have any doubt, they could always hide it in a bale of marijuana,” Allison half-joked.

  So Allison is mostly focused on the means—the ability of a terrorist group to acquire a nuclear weapon. If we want to reduce the risk of a nuclear version of 9/11, controlling the means would be the way.

  Experts believe there are about 20,000 nuclear warheads in the world today60—down from a peak of 65,000 in the 1980s. A threat could theoretically come from any of the nine countries that possess nuclear weapons today—even the United States has lost track of eleven of its nuclear weapons throughout its history61—and other countries may be trying to develop them. But Allison’s concern stems primarily from two nuclear states: Russia and Pakistan.

  In Allison’s view, the risk has lessened some in the former country. In part because of successful programs like the one sponsored by senators Sam Nunn and Richard Lugar, there are no longer active nuclear weapons in the outlying states of the former Soviet Union. And in Russia itself, the number of nuclear weapons has declined to 11,000 today from a peak of 30,000 in 1985.

  If the risk in Russia has been reduced, however, the threat posed by Pakistan has increased—perhaps markedly. “If you map weapons of mass destruction and terrorism, all the roads intersect in Pakistan,” Allison told me.

  Although Pakistan is ostensibly an ally of the United States, even the most generous interpretation would suggest that it represents a problem as well as a solution in the effort to contain terrorism. The country had initially been reluctant to cooperate with the United States after the September 11 attacks, and Pakistan’s president later claimed that the U.S. had resorted to a threat to bomb it “back to the stone age” before it complied.62 Osama bin Laden had been living in Abbottabad, Pakistan, for as many as six years63 before he was killed. Meanwhile, Pakistan has roughly one hundred nuclear weapons and is building additional nuclear facilities and delivery systems at a rapid pace.64 The country now ranks seventh in the world in the Economist’s Political Instability Index, up significantly from the recent past,65 meaning that the risk of a coup d’état or a revolution is quite high. A new regime could be openly hostile to the United States. All the conditions that a terrorist might need to acquire a nuclear weapon could then be in place.

  Because of the deteriorating situation in Pakistan, as well as the residual threat posed by other countries, Allison told me that he was still pretty much where he was in 2004—he envisions a better-than-even chance of a nuclear attack on the United States within the next decade unless there is a change in trajectory.

  Allison also has his critics, including Michael Levi, whom I visited at his office at the Council on Foreign Relation
s in New York. Like Aaron Clauset, Levi has an eccentric background: he studied theoretical physics at Princeton and was a technical consultant to the popular television program 24, which depicted a terrorist group attempting to detonate a nuclear weapon in Los Angeles.*

  Levi thinks there is some risk of an attack. “When I first came here,” he told me, “one of the first things I did was to draw the rings around Grand Central Station to see how a ten kiloton bomb would affect my apartment.” But he thinks Allison’s estimate is much too high, and questions several of his premises.

  For one thing, Levi thinks that Allison takes the motive of terrorist groups too much for granted. It’s not that Al Qaeda wouldn’t have the aspiration to blow up Manhattan, he says. But groups and individuals have all sorts of aspirations that they make no effort to act on because they doubt their ability to achieve them. Terrorist groups, in Levi’s view, place a high premium on the likelihood of their plots’ succeeding. A failed attempt would blow the group’s cover and invite unwanted attention from the United States and other governments. A failed attack could also harm the group’s credibility with both existing members and potential new recruits. Terrorist organizations are fundamentally weak and unstable: as is supposedly true of new restaurants, 90 percent of them fail within the first year.66 Their recruitment message relies, in large part, on convincing their members that they are able to deliver redemption from perceived injustices.67 Al Qaeda, in particular, had a very high success rate on the attacks it attempted up to and including September 11, something that may help to explain its uncommon longevity. But if a terrorist group’s ability to achieve that redemption is called into question, their recruits might go elsewhere.

  A nuclear attack would be hard to pull off. It is not that terrorist groups are inherently averse to intricate planning—the September 11 attacks were in the works for five years. But the more complicated the plot, the more cooperation is required from a larger number of participants, each of whom carries the risk of defecting from the group or being detected by counterterrorism authorities. A nuclear attack would also require significant and highly specialized technical knowledge—far more than four terrorists learning how to fly a 767. The pool of nuclear physicists is small to begin with, and the number who could be trusted to maintain loyalty to a terrorist group is smaller still.68 “If they found a guy who had an engineering degree from a university and put him in charge of the nuclear team,” Levi told me, “I wouldn’t expect much to come of that.”