Army of None
Page 49
287
“arbitrary for a decision to be taken”: Christof Heyns, interview, May 18, 2016.
288
“fundamental violation”: Peter Asaro, interview, December 19, 2016.
288
the right to die a dignified death in war: Of course, dying honorably has often been important in warrior culture, but that is not the same as extending the enemy the opportunity to die an honorable death.
288
“war without reflection is mechanical slaughter”: Nick Cumming-Bruce, “U.N. Expert Calls for Halt on Military Robots,” New York Times, May 30, 2013, sec. Europe, https://www.nytimes.com/2013/05/31/world/europe/united-nations-armed-robots.html.
290
“If you eliminate the moral burden”: Peter Asaro, interview, December 19, 2016.
290
“moral injury”: David Wood, “Moral Injury,” The Huffington Post, accessed June 17, 2017, http://projects.huffingtonpost.com/projects/moral-injury. Shira Maguen and Brett Litz, “Moral Injury in the Context of War,” PTSD: National Center for PTSD, accessed June 17, 2017, https://www.ptsd.va.gov/professional/co-occurring/moral_injury_at_war.asp. “What Is Moral Injury,” The Moral Injury Project, accessed June 17, 2017, http://moralinjuryproject.syr.edu/about-moral-injury/.
290
the most traumatic thing a soldier can experience: Grossman, On Killing, 87–93, 156–158.
293
“One of the places that we spend”: Paul Selva, “Innovation in the Department of Defense with General Paul Selva,” Center for Strategic and International Studies, https://www.csis.org/events/innovation-defense-department-general-paul-selva. Remarks on autonomous weapons begin around 39:00.
293
“Because we take our values to war”: Paul Selva, testimony before the Senate Armed Services Committee, July 18, 2017, https://www.armed-services.senate.gov/hearings/17-07-18-nomination_--selva. Comments on autonomous weapons begin around 1:11:10.
294
“War is about attempting to increase”: Jody Williams, interview, October 27, 2016.
294
“crosses a moral and ethical Rubicon”: Ibid.
294
“You know the difference between a good robot and a bad robot”: Ibid.
295
good Terminators: Ron Arkin, interview, June 8, 2016.
295
“beyond-IHL principle of human dignity”: Ken Anderson, interview, January 6, 2016.
295
“then we must ask ourselves whether”: Christof Heyns, interview, May 18, 2016.
295
“as long as we don’t lose our soul”: Ron Arkin, interview, June 8, 2016.
296
“Killing Japanese didn’t bother me very much”: “General Curtis E. LeMay, (1906–1990),” PBS.org, http://www.pbs.org/wgbh//amex/bomb/peopleevents/pandeAMEX61.html.
296
“I am tired and sick of war”: “William Tecumseh Sherman.”
18 Playing with Fire: Autonomous Weapons and Stability
298
further incentivized the Soviet Union: Michael S. Gerson, “The Origins of Strategic Stability: The United States and the Threat of Surprise Attack,” in Elbridge A. Colby and Michael S. Gerson, eds., Strategic Stability: Contending Interpretations (Carlisle, PA: Strategic Studies Institute and U.S. Army War College Press, 2013), 3–35.
298
“we have to worry about his striking us”: T. C. Schelling, Surprise Attack and Disarmament (Santa Monica, CA: RAND, December 10, 1958).
298
“when neither in striking first”: Ibid.
299
“In a stable situation”: Elbridge Colby, “Defining Strategic Stability: Reconciling Stability and Deterrence,” in Colby and Gerson, Strategic Stability, 57.
299
“War termination”: Thomas C. Schelling, Arms and Influence (New Haven and London: Yale University Press, 1966), 203–208. Fred Ikle, Every War Must End (New York: Columbia Classics, 2005).
299
offense-defense balance: Charles L. Glaser and Chaim Kaufmann, “What Is the Offense-Defense Balance and Can We Measure It? (Offense, Defense, and International Politics),” International Security 22, no. 4 (Spring 1998).
301
Outer Space Treaty: Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space, including the Moon and Other Celestial Bodies, 1967, http://www.unoosa.org/oosa/en/ourwork/spacelaw/treaties/outerspacetreaty.html.
301
Seabed Treaty: Treaty on the Prohibition of the Emplacement of Nuclear Weapons and Other Weapons of Mass Destruction on the Seabed and the Ocean Floor and in the Subsoil Thereof, 1971, https://www.state.gov/t/isn/5187.htm.
301
Environmental Modification Convention: Convention on the Prohibition of Military or Any Other Hostile Use of Environmental Modification Techniques, 1977, https://www.state.gov/t/isn/4783.htm.
301
Anti-Ballistic Missile (ABM) Treaty: Treaty Between The United States of America and The Union of Soviet Socialist Republics on The Limitation of Anti-Ballistic Missile Systems (ABM Treaty), 1972, https://www.state.gov/t/avc/trty/101888.htm.
301
Intermediate-Range Nuclear Forces (INF) Treaty: U.S. Department of State, “Treaty Between the United States of American and the Union of Soviet Socialist Republics on the Elimination of their Intermediate-Range and Shorter-Range Missiles,” accessed June 17, 2017, https://www.state.gov/www/global/arms/treaties/inf1.html#treaty.
301
neutron bombs: In practice, though, any adjustable dial-a-yield nuclear weapon could function as a neutron bomb.
301
attacker could use the conquered territory: This is not to say that neutron bombs did not have, in theory, legitimate uses. The United States viewed neutron bombs as valuable because they could be used to defeat Soviet armored formations on allied territory without leaving residual radiation.
302
U.S. plans to deploy neutron bombs to Europe: Mark Strauss, “Though It Seems Crazy Now, the Neutron Bomb Was Intended to Be Humane,” io9, September 19, 2014, http://io9.gizmodo.com/though-it-seems-crazy-now-the-neutron-bomb-was-intende-1636604514.
302
first-mover advantage in naval warfare: Wayne Hughes, Fleet Tactics and Coastal Combat, 2nd ed. (Annapolis, MD: Naval Institute Press, 1999).
302
“Artificial Intelligence, War, and Crisis Stability”: Michael C. Horowitz, “Artificial Intelligence, War, and Crisis Stability,” November 29, 2016. [unpublished manuscript, as of June 2017].
303
robot swarms will lead to: Jean-Marc Rickli, “Some Considerations of the Impact of LAWS on International Security: Strategic Stability, Non-State Actors and Future Prospects,” paper submitted to Meeting of Experts on Lethal Autonomous Weapons Systems, United Nations Convention on Certain Conventional Weapons (CCW), April 16, 2015, http://www.unog.ch/80256EDD006B8954/(httpAssets)/B6E6B974512402BEC1257E2E0036AAF1/$file/2015_LAWS_MX_Rickli_Corr.pdf.
303
lower the threshold for the use of force: Peter M. Asaro, “How Just Could a Robot War Be?” http://peterasaro.org/writing/Asaro%20Just%20Robot%20War.pdf.
303
“When is it that you would deploy these systems”: Michael Horowitz, interview, December 7, 2016.
305
“The premium on haste”: Schelling, Arms and Influence, 227.
305
“when speed is critical”: Ibid, 227.
305
control escalation: See also Jürgen Altmann and Frank Sauer, “Autonomous Weapon Systems and Strategic Stability,” Survival 59:5 (2017), 117–42.
305
“restraining devices for weapons”: Ibid, 231.
306
war among nuclear powers: Elbridge Colby, “America Must Prepare for Limited War,” The National Interest, October 21, 2015, http://nationalinterest.org/feature/america-must-prepare-limited-war-14104.
306
“[S]tates [who employ autonomous weapons]”: Michael Carl Haas, “Autonomous Weapon Systems: The Military’s Smartest Toys?” The National Interest, November 20, 2014, http://nationalinterest.org/feature/autonomous-weapon-systems-the-militarys-smartest-toys-11708.
307
test launch of an Atlas ICBM: Sagan, The Limits of Safety, 78–80.
307
U-2 flying over the Arctic Circle: Martin J. Sherwin, “The Cuban Missile Crisis at 50,” Prologue Magazine 44, no. 2 (Fall 2012), https://www.archives.gov/publications/prologue/2012/fall/cuban-missiles.html.
307
General Thomas Power: Powers’ motivations for sending this message have been debated by historians. See Sagan, The Limits of Safety, 68–69.
308
“commander’s intent”: Headquarters, Department of the Army, Field Manual 100-5 (June 1993), 6-6.
308
“If . . . the enemy commander has”: Lawrence G. Shattuck, “Communicating Intent and Imparting Presence,” Military Review (March–April 2000), http://www.au.af.mil/au/awc/awcgate/milreview/shattuck.pdf, 66.
308
“relocating important assets”: Haas, “Autonomous Weapon Systems.”
309
“strategic corporal”: Charles C. Krulak, ““The Strategic Corporal: Leadership in the Three Block War,” Marines Magazine (January 1999).
309
“affirmative human decision”: “No agency of the Federal Government may plan for, fund, or otherwise support the development of command control systems for strategic defense in the boost or post-boost phase against ballistic missile threats that would permit such strategic defenses to initiate the directing of damaging or lethal fire except by affirmative human decision at an appropriate level of authority.” 10 U.S.C. 2431 Sec. 224.
310
“you still have the problem that that’s”: David Danks, interview, January 13, 2017.
310
“[B]efore we sent the U-2 out”: Robert McNamara, Interview included as special feature on Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (DVD). Columbia Tristar Home Entertainment, (2004) [1964].
311
“We’re going to blast them now!”: William Burr and Thomas S. Blanton, eds., “The Submarines of October,” The National Security Briefing Book, No. 75, October 31, 2002, http://nsarchive.gwu.edu/NSAEBB/NSAEBB75/. “The Cuban Missile Crisis, 1962: Press Release, 11 October 2002, 5:00 PM,” accessed June 17, 2017, http://nsarchive.gwu.edu/nsa/cuba_mis_cri/press3.htm.
311
game of chicken: Schelling, Arms and Influence, 116–125. Herman Kahn, On Escalation: Metaphors and Scenarios (New Brunswick and London: Transaction Publishers, 2010), 10-11.
311
“takes the steering wheel”: Kahn, On Escalation, 11.
312
“how would the Kennedy Administration”: Horowitz, “Artificial Intelligence, War, and Crisis Stability,” November.
312
“because of the automated and irrevocable”: Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb, directed by Stanley Kubrick (1964).
313
“Dead Hand”: Nicholas Thompson, “Inside the Apocalyptic Soviet Doomsday Machine,” WIRED, September 21, 2009, https://www.wired.com/2009/09/mf-deadhand/. Vitalii Leonidovich Kataev, interviewed by Ellis Mishulovich, May 1993, http://nsarchive.gwu.edu/nukevault/ebb285/vol%20II%20Kataev.PDF. Varfolomei Vlaimirovich Korobushin, interviewed by John G. Hines, December 10, 1992, http://nsarchive.gwu.edu/nukevault/ebb285/vol%20II%20Korobushin.PDF.
313
Accounts of Perimeter’s functionality differ: Some accounts by former Soviet officials state that the Dead Hand was investigated and possibly even developed, but never deployed operationally. Andrian A. Danilevich, interview by John G. Hines, March 5, 1990, http://nsarchive.gwu.edu/nukevault/ebb285/vol%20iI%20Danilevich.pdf, 62-63; and Viktor M. Surikov, interview by John G. Hines, September 11, 1993, http://nsarchive.gwu.edu/nukevault/ebb285/vol%20II%20Surikov.PDF, 134-135. It is unclear, though, whether this refers in reference or not to a fully automatic system. Multiple sources confirm the system was active, although the degree of automation is ambiguous in their accounts: Kataev, 100–101; and Korobushin, 107.
313
remain inactive during peacetime: Korobushin, 107; Thompson, “Inside the Apocalyptic Soviet Doomsday Machine.”
313
network of light, radiation, seismic, and pressure: Ibid.
313
leadership would be cut out of the loop: Ibid.
313
rockets that would fly over Soviet territory: Kataev. Korobushin.
314
Perimeter is still operational: Thompson, “Inside the Apocalyptic Soviet Doomsday Machine.”
314
“stability-instability paradox”: Michael Krepon, “The Stability-Instability Paradox, Misperception, and Escalation Control in South Asia,” The Stimson Center, 2003, https://www.stimson.org/sites/default/files/file-attachments/stability-instability-paradox-south-asia.pdf. B.H. Liddell Hart, Deterrent or Defence (London: Stevens and Sons, 1960), 23.
315
“madman theory”: Harry R. Haldeman and Joseph Dimona, The Ends of Power (New York: Times Books, 1978), 122. This is not a new idea. It dates back at least to Machiavelli. Niccolo Machiavelli, Discourses on Livy, Book III, Chapter 2.
315
“the threat that leaves something to chance”: Thomas C. Schelling, The Strategy of Conflict (Cambridge, MA: Harvard University Press, 1960).
316
“There’s a real problem here”: David Danks, interview, January 13, 2017.
316
“Autonomous weapon systems are very new”: Ibid.
316
“it’s just completely unreasonable”: Ibid.
317
“Mr. President, we and you ought not”: Department of State Telegram Transmitting Letter From Chairman Khrushchev to President Kennedy, October 26, 1962, http://microsites.jfklibrary.org/cmc/oct26/doc4.html.
317
“there are scenarios in which”: Haas, “Autonomous Weapon Systems.”
19 Centaur Warfighters: Humans + Machines
321
Gary Kasparov: Mike Cassidy, “Centaur Chess Brings out the Best in Humans and Machines,” BloomReach, December 14, 2014, http://bloomreach.com/2014/12/centaur-chess-brings-best-humans-machines/.
321
centaur chess: Tyler Cowen, “What are Humans Still Good for? The Turning Point in Freestyle Chess may be Approaching,” Marginal Revolution, November 5, 2013, http://marginalrevolution.com/marginalrevolution/2013/11/what-are-humans-still-good-for-the-turning-point-in-freestyle-chess-may-be-approaching.html.
322
“On 17 April 1999”: Mike Pietrucha, “Why the Next Fighter will be Manned, and the One After That,” War on the Rocks, August 5, 2015, http://warontherocks.com/2015/08/why-the-next-fighter-will-be-manned-and-the-one-after-that/.
323
Commercial airliners use automation: Mary Cummings and Alexander Stimpson, “Full Auto Pilot: Is it Really Necessary to Have a Human in the Cockpit?,” Japan Today, May 20, 2015, http://www.japantoday.com/category/opinions/view/full-auto-pilot-is-it-really-necessary-to-have-a-human-in-the-cockpit.
324
“Do Not Engage Sector”: Mike Van Rassen, “Counter-Rocket, Artillery, Mortar (C-RAM),” Program Executive Office Missiles and Space, accessed June 16, 2017, Slide 28, http://www.msl.army.mil/Documents/Briefings/C-RAM/C-RAM%20Program%20Overview.pdf.
324
“The human operators do not aim”: Sam Wallace, “The Proposed Ban on Offensive Autonomous Weapons is Unrealistic and Dangerous,” Kurzweilai, August 5, 2015, http://www.kurzweilai.net/the-proposed-ban-on-offensive-autonomous-weapons-is-unrealistic-and-dangerous.
325
“unwarranted and uncritical trust”: Hawley, “Not by Widgets Alone.”
325
the human does not add any value: Giv
en recent advances in machine learning, it is possible we are at this point now. In December 2017, the AI research company DeepMind unveiled AlphaZero, a single algorithm that had achieved superhuman play in chess, go, and the Japanese strategy game shogi. Within a mere four hours of self-play and with no training data, AlphaZero eclipsed the previous top chess program. The method behind AlphaZero, deep reinforcement learning, appears to be so powerful that it is unlikely that humans can add any value as members of a “centaur” human-machine team for these games. Tyler Cowen, “The Age of the Centaur Is *Over* Skynet Goes Live,” MarginalRevolution.com, December 7, 2017, http://marginalrevolution.com/marginalrevolution/2017/12/the-age-of-the-centaur-is-over.html. David Silver et al., “Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm,” December 5, 2017, https://arxiv.org/pdf/1712.01815.pdf.
325
as computers advance: Cowen, “What Are Humans Still Good For?”
327
jam-resistant communications: Sayler, “Talk Stealthy to Me.” Paul Scharre, “Yes, Unmanned Aircraft Are The Future,” War on the Rocks, August 11, 2015, https://warontherocks.com/2015/08/yes-unmanned-combat-aircraft-are-the-future/.Amy Butler, “5th-To-4th Gen Fighter Comms Competition Eyed In Fiscal 2015,” Aviation Week Network, June 18, 2014, http://aviationweek.com/defense/5th-4th-gen-fighter-comms-competition-eyed-fiscal-2015.
330
“What application are we trying”: Peter Galluch, interview, July 15, 2016.
330
“If they don’t exist, there is no”: Jody Williams, interview, October 27, 2016.
330
“mutual restraint”: Evan Ackerman, “We Should Not Ban ‘Killer Robots,’ and Here’s Why,” IEEE Spectrum: Technology, Engineering, and Science News, July 29, 2015, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/we-should-not-ban-killer-robots.
20 The Pope and the Crossbow: The Mixed History of Arms Control
331
“The key question for humanity today”: “Autonomous Weapons: An Open Letter From AI & Robotics Researchers.”
332
Whether or not a ban succeeds: For analysis of why some weapons bans work and some don’t, see: Rebecca Crootof, http://isp.yale.edu/sites/default/files/publications/killer_robots_are_here_final_version.pdf; Sean Watts, “Autonomous Weapons: Regulation Tolerant or Regulation Resistant?” Temple International & Comparative Law Journal 30 (1), Spring 2016, 177–187; and Rebecca Crootof, https://www.lawfareblog.com/why-prohibition-permanently-blinding-lasers-poor-precedent-ban-autonomous-weapon-systems.