Army of None

by Paul Scharre

  Horowitz asked whether autonomous weapons might excel in such a situation. Here, machines’ rigid adherence to rules and lack of recallability would be a benefit. A robot designed to never swerve would be the perfect driver to win at chicken. In “Artificial Intelligence, War, and Crisis Stability,” Horowitz presented the thought experiment of an alternative Cuban Missile Crisis in which the U.S. ships conducting the blockade were autonomous weapons. They would be programmed to fire on any Soviet ships crossing the blockade line. If this could be credibly communicated to the Soviets, it would have put the onus of avoiding conflict on the Soviets. The problem, Horowitz asked, was “how would the Kennedy Administration have persuaded the Soviet Union that that was the case?” There would be no way to convincingly prove to Soviet leadership that the robotic vessels were actually programmed to fire. U.S. leaders could claim that was the case, but the claim would be meaningless, since that’s also what they would say if they were bluffing. The United States would certainly not allow the Soviets to inspect the code of U.S. ships at the blockade. There would be no credible way to demonstrate that one had, in fact, tied one’s hands. It would be the equivalent of ripping out the steering wheel, but being unable to throw it out the window. (Similarly, the Soviets could program their ships to run the blockade without any option for turning back, but there would be no way to prove to the Americans they had done so.)

  Stanley Kubrick’s 1964 film Dr. Strangelove explores the bizarre logic of deterrence and mutual assured destruction. In the film, the Soviet ambassador explains to an assembled group of American military and political leaders that the Soviet Union has built a “doomsday machine” which, if the Soviet Union is attacked, will automatically launch a massive nuclear counterattack that will destroy humanity. The title character Dr. Strangelove explains, “because of the automated and irrevocable decision-making process, which rules out human meddling, the doomsday machine is terrifying . . . . and completely capable and convincing.” Unfortunately, in the movie, the Soviets fail to tell their American counterparts that they have built such a device. Strangelove yells at the Soviet ambassador, “the whole point of the doomsday machine is lost if you keep it a secret!”

  As an example of truth being sometimes stranger than fiction, after the Cold War evidence emerged that the Soviets did in fact build a semiautomatic doomsday device, nicknamed “Dead Hand.” Officially called “Perimeter,” the system was reportedly an automated nuclear command-and-control system designed to allow a massive retaliatory attack even if a U.S. first strike took out Soviet leadership. Accounts of Perimeter’s functionality differ, but the essential idea was that the system would remain inactive during peacetime but, in the event of a crisis, it could be activated as a “fail-deadly” mechanism for ensuring retaliation. When active, a network of light, radiation, seismic, and pressure sensors would evaluate whether there had been any nuclear detonations on Soviet soil. If a nuclear detonation was detected, then the system would check for communications to the General Staff of the Soviet military. If communications were active, then it would wait a predetermined about of time, ranging from on the order of fifteen minutes to an hour, for an order to cancel the launch. If there was no order to stop the launch, Perimeter would act like a “dead man’s switch,” a switch that is automatically triggered if a person becomes incapacitated or dies. In many hazardous machines, a dead man’s switch is used as a fail-safe mechanism. If a person becomes incapacitated, the machine will revert to a safe mode of operation, like a lawnmower shutting off if you release the handle. In this case, Perimeter was intended to “fail deadly.” If there was no signal from the Soviet General Staff to halt the launch, Perimeter would bypass normal layers of command and transfer launch authority directly to individuals within a deep underground protected bunker. There would still be a human in the loop, but the decision would reside with whichever staff officer was on duty at the time. Soviet leadership would be cut out of the loop. With a push of a button, that individual could launch a series of communications rockets that would fly over Soviet territory and beam down the nuclear launch codes to missiles in hardened silos. Soviet ICBMs would then launch a massive strike on the United States, the last zombie attack of a dying nation.

  There was a purpose to the madness. In theory, if everything worked properly, a system like Perimeter would enhance stability. Because a retaliatory strike would be assured, the system would remove the need for haste from Soviet leaders’ decision-making in a crisis. If there were warnings of an incoming U.S. surprise attack, as was the case in 1983 in the Stanislav Petrov incident and again in 1995 when Russian military leaders brought the nuclear suitcase to Boris Yeltsin in response to a Norwegian scientific rocket launch, there would be no rush to respond. Soviet or Russian leaders would have no incentive to fire their nuclear missiles in an ambiguous situation, because even if the United States succeeded in a decapitating strike, retaliation was assured. The knowledge of this would also presumably deter the United States from even considering a preemptive first strike. The problem, of course, is that such a system comes with tremendous risks. If Perimeter were to falsely detect an event, as the Soviet Oko satellite system did in 1983 when it falsely detected U.S. ICBM launches, or if Soviet leaders were unable to stop the mechanism once it was activated, the system would obliterate humanity.

  By some accounts, Perimeter is still operational within Russia today.

  STABILITY-INSTABILITY PARADOX AND THE MAD ROBOT THEORY

  The logic of mutual assured destruction (MAD) is to make any nuclear attack inherently suicidal. If a retaliatory response is assured, then attacking the enemy is akin to attacking oneself. This dynamic is stabilizing in the sense that it deters both sides from using nuclear weapons. Ironically, though, over time strategists began to worry that too much stability was a bad thing. This became known as the “stability-instability paradox.”

  The essence of the problem is that if nuclear weapons are fully and mutually restrained, then they could lose their value as a deterrent. This could embolden aggression below the nuclear threshold, since countries could be confident that an adversary would not respond with nuclear weapons. Under this logic, some instability—some risk of accidents and miscalculation—is a good thing, because it induces caution. Returning to the gunslingers in their standoff, if stabilizing measures are those that make it less likely that a gunslinger will draw his weapon, too much stability might encourage other forms of aggression. One might be willing to insult or even steal from the other gunslinger, confident that he wouldn’t draw his gun, since doing so would be suicidal.

  One response to this paradox is the “madman theory.” As the acronym MAD implies, the logic of mutual assured destruction is fundamentally insane. Only a mad person would launch a nuclear weapon. The principle behind the madman theory, espoused by President Richard Nixon, is to convince the enemy’s leadership that a nation’s leaders are so volatile and irrational that they just might push the button. Mutual suicide or no, one would hesitate to insult a gunslinger with a reputation for rash, even self-destructive acts.

  This suggests another way autonomous weapons might improve stability: the “mad robot theory.” If countries perceive autonomous weapons as dangerous, as introducing an unpredictable element into a crisis that cannot be completely controlled, then introducing them into a crisis might induce caution. It would be the equivalent of what Thomas Schelling has described as “the threat that leaves something to chance.” By deploying autonomous weapons into a tense environment, a country would effectively be saying to the enemy, “Things are now out of my hands. Circumstances may lead to war; they may not. I cannot control it, and your only course of action if you wish to avoid war is to back down.” Unlike the problem of credibly tying one’s hands by locking in escalatory rules of engagement, this threat does not require convincing the enemy what the autonomous weapons’ rules of engagement are. In fact, uncertainty makes the “mad robot” threat more credible, since deterrence hinges on the robot’s
unpredictability, rather than the certainty of its actions. Deploying an untested and unverified autonomous weapon would be even more of a deterrent, since one could convincingly say that its behavior was truly unpredictable.

  What is interesting about this idea is that its efficacy rests solely on humans’ perception of autonomous weapons, and not the actual functionality of the weapons themselves. The weapons may be reliable or unreliable—it doesn’t matter. What matters is that they are perceived as unpredictable and, as a result, induce caution. Of course, the actual functionality of the weapons does matter when it comes to how a crisis unfolds. The key is the difference between how humans perceive the risk of autonomous weapons and their actual risk. If leaders overestimate the risks of autonomous weapons, then there is nothing to worry about. Their introduction into crises will induce caution but they will be unlikely to cause harm. If leaders underestimate their risks, then their use invites disaster.

  How accurately people can assess the risks of autonomous weapons hinges on individual psychology and how organizations evaluate risk in complex systems. I asked David Danks what he thought about peoples’ ability to accurately assess these risks, and his answer was not encouraging. “There’s a real problem here for autonomous weapons,” he said. For starters, he explained that people are poor predictors of behavior in systems that have feedback loops, where one action creates a counterreaction and so on. (Real-world experience with complex autonomous systems in uncontrolled environments, such as stock trading, lends weight to this theory.)

  Furthermore, Danks said, due to projection bias, people are poor predictors of risk for situations for which they have no experience. For example, Danks explained, people are good estimators of their likelihood of getting into an automobile accident because they frequently ride in vehicles. But when they have no prior knowledge, then their ability to accurately assess risks falls apart. “Autonomous weapon systems are very new. They aren’t just a bigger gun,” he said. “If you think of them as a bigger gun, you say, ‘Well we’ve got a lot of experience with guns.’” That might lead one to think that one could accurately evaluate the risks of autonomous weapons. But Danks said he thought they were “qualitatively different” than other weapons.

  This suggests we lack the necessary experience to accurately assess the risks of autonomous weapons. How much testing is required to ensure an autonomous weapon fails less than 0.0001 percent of the time? We don’t know, and we can’t know until we build up experience with more sophisticated autonomous systems over time. Danks said it would be different if we already had extensive experience with safely operating complex autonomous systems in real-world environments. Unfortunately, the experience we do have suggests that surprises are often lurking below the surface of complex systems. Danks concluded that “it’s just completely unreasonable and hopelessly optimistic to think that we would be good at estimating the risks.”

  UNTYING THE KNOT

  At the height of the Cuban Missile Crisis, Soviet Premier Nikita Khrushchev sent an impassioned letter to President Kennedy calling on them to work together to step back from the brink of nuclear war:

  Mr. President, we and you ought not now to pull on the ends of the rope in which you have tied the knot of war, because the more the two of us pull, the tighter that knot will be tied. And a moment may come when that knot will be tied so tight that even he who tied it will not have the strength to untie it, and then it will be necessary to cut that knot, and what that would mean is not for me to explain to you, because you yourself understand perfectly of what terrible forces our countries dispose.

  Autonomous weapons raise troubling concerns for stability and escalation control in crises. Michael Carl Haas concluded, “there are scenarios in which the introduction of autonomous strike systems could result in temporary loss of high-level control over operations, and unwanted escalation (conventional or nuclear).” He argued policymakers “should exercise prudence and caution” before adding autonomous weapon systems “into an equation that is highly complex as it stands.” Their rigid rule-following could tighten the knot, with no understanding of the context for or consequences of their actions.

  During the Cuban Missile Crisis, U.S. leaders were constantly trying to understand the psychology of their Soviet counterparts. While they had differing interests, President Kennedy empathized with Premier Khrushchev’s position. Kennedy understood that if the United States moved against Cuba, the Soviets would be compelled to respond elsewhere in the world, perhaps in Berlin. Kennedy understood that he needed to give Khrushchev an option to remove the missiles from Cuba while saving face. Kennedy and others were able to think through the second- and third-order consequences of their actions. (Khrushchev eventually agreed to remove the missiles in exchange for a U.S. pledge not to invade Cuba and a secret promise to remove American missiles from Turkey.) Vasili Arkhipov, on the Soviet submarine B-59, similarly understood that if they fired a nuclear torpedo, obliterating a U.S. aircraft carrier, the Americans would feel compelled to respond with nuclear weapons elsewhere. The result would be escalating to a level it might be impossible to back down from.

  Humans are not perfect, but they can empathize with their opponents and see the bigger picture. Unlike humans, autonomous weapons would have no ability to understand the consequences of their actions, no ability to step back from the brink of war. Autonomous weapons would not take away all human decision-making in crises, but they do have the potential to tighten the knot, perhaps so far that it cannot be undone.

  * “Strategic stability” is often used to refer specifically to nuclear weapons—hence, the use of “stability” in this book when used in reference to autonomous weapons.

  PART VI

  Averting Armageddon: The Weapon of Policy

  19

  CENTAUR WARFIGHTERS

  HUMANS + MACHINES

  If there is one common theme across the legal, ethical, and strategic issues surrounding autonomous weapons, it is whether, and how much, the decision to use force depends on context. Machines, at least for the foreseeable future, will not be as good as humans at understanding the context for their actions. Yet there are circumstances in which machines perform far better than humans. The best decision-making system would be one that leverages the advantages of each. Hybrid human-machine cognitive systems, often called “centaur warfighters” after the classic Greek myth of the half-human, half-horse creature, can leverage the precision and reliability of automation without sacrificing the robustness and flexibility of human intelligence.

  THE CENTAUR EDGE

  To glimpse the future of cognition, we need look no further than one of the most high-profile areas in which AI has bested humans: chess. In 1997, IBM’s Deep Blue defeated world chess champion Gary Kasparov, cementing the reality that humans are no longer the best chess players in the world. But neither, as it turns out, are machines. A year later, Kasparov founded the field of “advanced chess,” also called centaur chess, in which humans and AI cooperate on the same team. The AI can analyze possible moves and identify vulnerabilities or opportunities the human player might have missed, resulting in blunder-free games. The human player can manage strategy, prune AI searches to focus on the most promising areas, and manage differences between multiple AIs. The AI system gives feedback to the human player, who then decides what move to make. By leveraging the advantages of human and machine, centaur chess results in a better game than either humans or AI can achieve on their own.

  Understanding how this might work for weapons engagements requires first disaggregating the various roles a human performs today in targeting decisions: (1) acting as an essential operator of the weapon system; (2) acting as a fail-safe; and (3) acting as a moral agent.

  When acting as an “essential operator” of the weapon system, a human performs a vital function without which the weapon cannot work. A human “painting” a target with a laser to direct a laser-guided bomb onto the target is acting as an essential operator. When a human acts
as a fail-safe, the weapon could function on its own, but the human is in the loop as a backup to intervene if it fails or if circumstances change such that the engagement is no longer appropriate. When acting as a “moral agent,” the human is making value-based judgments about whether the use of force is appropriate.

  An anecdote from the U.S. air campaign over Kosovo in 1999 includes an instructive example of all three roles in action simultaneously:

  On 17 April 1999, two F-15E Strike Eagles, Callsign CUDA 91 and 92, were tasked to attack an AN/TPS-63 mobile early warning radar located in Serbia. The aircraft carried AGM-130, a standoff weapon that is actually remotely flown by the weapons system officer (WSO) in the F-15E, who uses the infra-red sensor in the nose of the weapon to detect the target. CUDA 91 . . . launched on coordinates provided by the Air Operations Center. As the weapon approached the suspected target location, the crew had not yet acquired the [enemy radar]. At 12 seconds from impact, the picture became clearer. . . . [The pilots saw the profile outline of what appeared to be a church steeple.] Three seconds [from impact], the WSO makes the call: “I’m ditching in this field” and steers the weapon into an empty field several hundred meters away. . . . Postflight review of the tape revealed no object that could be positively identified as a radar, but the profile of a Serbian Orthodox church was unmistakable.