by Paul Scharre
208
Syria shot down: Missy Ryan, “U.S. Drone Believed Shot down in Syria Ventured into New Area, Official Says,” Washington Post, March 19, 2015, https://www.washingtonpost.com/world/national-security/us-drone-believed-shot-down-in-syria-ventured-into-new-area-official-says/2015/03/19/891a3d08-ce5d-11e4-a2a7-9517a3a70506_story.html.
208
Turkey shot down: “Turkey Shoots down Drone near Syria, U.S. Suspects Russian Origin,” Reuters, October 16, 2015, http://www.reuters.com/article/us-mideast-crisis-turkey-warplane-idUSKCN0SA15K20151016.
209
China seized a small underwater robot: “China to Return Seized U.S. Drone, Says Washington ‘Hyping Up’ Incident,” Reuters, December 18, 2016, http://www.reuters.com/article/us-usa-china-drone-idUSKBN14526J.
209
Navy Fire Scout drone: Elisabeth Bumiller, “Navy Drone Wanders Into Restricted Airspace Around Washington,” New York Times, August 25, 2010, https://www.nytimes.com/2010/08/26/us/26drone.html.
209
Army Shadow drone: Alex Horton, “Questions Hover over Army Drone’s 630-Mile Odyssey across Western US,” Stars and Stripes, March 1, 2017, https://www.stripes.com/news/questions-hover-over-army-drone-s-630-mile-odyssey-across-western-us-1.456505#.WLby0hLyv5Z.
209
RQ-170: Greg Jaffe and Thomas Erdbrink, “Iran Says It Downed U.S. Stealth Drone; Pentagon Acknowledges Aircraft Downing,” Washington Post, December 4, 2011, https://www.washingtonpost.com/world/national-security/iran-says-it-downed-us-stealth-drone-pentagon-acknowledges-aircraft-downing/2011/12/04/gIQAyxa8TO_story.html.
209
Reports swirled online: Scott Peterson and Payam Faramarzi, “Exclusive: Iran Hijacked US Drone, Says Iranian Engineer,” Christian Science Monitor, December 15, 2011, http://www.csmonitor.com/World/Middle-East/2011/1215/Exclusive-Iran-hijacked-US-drone-says-Iranian-engineer.
209
“complete bullshit”: David Axe, “Did Iran Hack a Captured U.S. Stealth Drone?” WIRED, April 24, 2012, https://www.wired.com/2012/04/iran-drone-hack/.
209
United States did awkwardly confirm: Bob Orr, “U.S. Official: Iran Does Have Our Drone,” CBS News, December 8, 2011, http://www.cbsnews.com/news/us-official-iran-does-have-our-drone/.
210
“networks of systems”: Heather Roff, interview, October 26, 2016.
210
“If my autonomous agent”: Ibid.
210
“What are the unexpected side effects”: Bradford Tousley, interview, April 27, 2016.
210
“I don’t know that large-scale military impacts”: Ibid.
210
“machine speed . . . milliseconds”: Ibid.
14 The Invisible War: Autonomy in Cyberspace
212
Internet Worm of 1988: Ted Eisenberg et al., “The Cornell Commission: On Morris and the Worm,” Communications of the ACM 32, 6 (June 1989), 706–709, http://www.cs.cornell.edu/courses/cs1110/2009sp/assignments/a1/p706-eisenberg.pdf;
212
over 70,000 reported cybersecurity incidents: Government Accountability Office, “Information Security: Agencies Need to Improve Controls over Selected High-Impact Systems,” GAO-16-501, Washington, DC, May 2016, http://www.gao.gov/assets/680/677293.pdf.
212
most frequent and most serious attacks: Ibid, 11.
212
exposed security clearance investigation data: James Eng, “OPM Hack: Government Finally Starts Notifying 21.5 Million Victims,” NBC News, October 1, 2015, http://www.nbcnews.com/tech/security/opm-hack-government-finally-starts-notifying-21-5-million-victims-n437126. “Why the OPM Hack Is Far Worse Than You Imagine,” Lawfare, March 11, 2016, https://www.lawfareblog.com/why-opm-hack-far-worse-you-imagine.
212
Chinese government claimed: David E. Sanger, “U.S. Decides to Retaliate Against China’s Hacking,” New York Times, July 31, 2015, https://www.nytimes.com/2015/08/01/world/asia/us-decides-to-retaliate-against-chinas-hacking.html. Sean Gallagher, “At First Cyber Meeting, China Claims OPM Hack Is ‘criminal Case’ [Updated],” Ars Technica, December 3, 2015, https://arstechnica.com/tech-policy/2015/12/at-first-cyber-meeting-china-claims-opm-hack-is-criminal-case/. David E. Sanger and Julie Hirschfeld Davis, “Hacking Linked to China Exposes Millions of U.S. Workers,” New York Times, June 4, 2015, https://www.nytimes.com/2015/06/05/us/breach-in-a-federal-computer-system-exposes-personnel-data.html.
212
affected Estonia’s entire electronic infrastructure: Dan Holden, “Estonia, Six Years Later,” Arbor Networks, May 16, 2013, https://www.arbornetworks.com/blog/asert/estonia-six-years-later/.
213
over a million botnet-infected computers: “Hackers Take Down the Most Wired Country in Europe,” WIRED, accessed June 14, 2017, https://www.wired.com/2007/08/ff-estonia/. “Denial-of-Service: The Estonian Cyberwar and Its Implications for U.S. National Security,” International Affairs Review, accessed June 14, 2017, http://www.iar-gwu.org/node/65.
213
“disastrous” consequences if Estonia removed the monument: “Hackers Take Down the Most Wired Country in Europe.”
213
Russian Duma official confirmed: “Russia Confirms Involvement with Estonia DDoS Attacks,” SC Media US, March 12, 2009, https://www.scmagazine.com/news/russia-confirms-involvement-with-estonia-ddos-attacks/article/555577/.
213
many alleged or confirmed cyberattacks: “Estonia, Six Years Later.”
213
cyberattacks against Saudi Arabia and the United States: Keith B. Alexander, “Prepared Statement of GEN (Ret) Keith B. Alexander* on the Future of Warfare before the Senate Armed Services Committee,” November 3, 2015, http://www.armed-services.senate.gov/imo/media/doc/Alexander_11-03-15.pdf.
213
team of professional hackers months if not years: David Kushner, “The Real Story of Stuxnet,” IEEE Spectrum: Technology, Engineering, and Science News, February 26, 2013, http://spectrum.ieee.org/telecom/security/the-real-story-of-stuxnet.
213
“zero days”: Kim Zetter, “Hacker Lexicon: What Is a Zero Day?,” WIRED, November 11, 2014, https://www.wired.com/2014/11/what-is-a-zero-day/.
213
Stuxnet had four: Michael Joseph Gross, “A Declaration of Cyber War.” Vanity Fair, March 2011, https://www.vanityfair.com/news/2011/03/stuxnet-201104.
214
programmable logic controllers: Gross, “A Declaration of Cyber War.” Nicolas Falliere, Liam O Murchu, and Eric Chien, “W32.Stuxnet Dossier,” Symantec Security Response, February 2011, https://www.symantec.com/content/en/us/enterprise/media/security_response/whitepapers/w32_stuxnet_dossier.pdf.
214
two encrypted “warheads”: Gross, “A Declaration of Cyber War.”
214
Computer security specialists widely agree: Falliere et al., “W32.Stuxnet Dossier,” 2, 7.
214
Natanz nuclear enrichment facility: Gross, “A Declaration of Cyber War.” Ralph Langner, “Stuxnet Deep Dive,” S4x12, https://vimeopro.com/s42012/s4-2012/video/35806770. Kushner, imeopro.com/s42012/Stuxnet.t
214
Nearly 60 percent of Stuxnet infections: Falliere et al., “W32.Stuxnet Dossier,” 5–7. Kim Zetter, “An Unprecedented Look at Stuxnet, the World’s First Digital Weapon,” WIRED, November 3, 2014, https://www.wired.com/2014/11/countdown-to-zero-day-stuxnet/.
214
sharp decline in the number of centrifuges: John Markoff and David E. Sanger, “In a Computer Worm, a Possible Biblical Clue,” New York Times, September 24, 2010, http://www.nytimes.com/2010/09/30/world/middleeast/30worm.html.
214
Security specialists have further speculated: Ibid. Gross, “A Declaration of Cyber War.”
215
“While attackers could control Stuxnet”: Falliere et al., “W32.Stuxnet Dossier,” 3.
215
“collateral damage”: Ibid, 7.
215
spread via USB to only three other machines: Ibid, 10.
215
self-terminate date: Ibid, 18.
215
Some experts saw these features as further evidence: Gross, “A Declaration of Cyber War.”
215
“open-source weapon”: Patrick Clair, “Stuxnet: Anatomy of a Computer Virus,” video, 2011, https://vimeo.com/25118844.
215
blueprint for cyber-weapons to come: Josh Homan, Sean McBride, and Rob Caldwell, “IRONGATE ICS Malware: Nothing to See Here . . . Masking Malicious Activity on SCADA Systems,” FireEye, June 2, 2016, https://www.fireeye.com/blog/threat-research/2016/06/irongate_ics_malware.html.
216
“It should be automated”: Keith B. Alexander, Testimony on the Future of Warfare, Senate Armed Services Committee, November 3, 2015, http://www.armed-services.senate.gov/hearings/15-11-03-future-of-warfare. Alexander’s comments on automation come up in the question-and-answer period, starting at 1:14:00.
217
DARPA held a Robotics Challenge: DARPA, “DARPA Robotics Challenge (DRC),” accessed June 14, 2017, http://www.darpa.mil/program/darpa-robotics-challenge. DARPA, “Home | DRC Finals,” accessed June 14, 2017, http://archive.darpa.mil/roboticschallenge/.
217
“automatically check the world’s software”: David Brumley, “Why CGC Matters to Me,” ForAllSecure, July 26, 2016, https://forallsecure.com/blog/2016/07/26/why-cgc-matters-to-me/.
217
“fully autonomous system for finding and fixing”: David Brumley, “Mayhem Wins DARPA CGC,” ForAllSecure, August 6, 2016, https://forallsecure.com/blog/2016/08/06/mayhem-wins-darpa-cgc/.
217
vulnerability is analogous to a weak lock: David Brumley, interview, November 24, 2016.
218
“There’s grades of security”: Ibid.
218
“an autonomous system that’s taking all of those things”: Ibid.
218
“Our goal was to come up with a skeleton key”: Ibid.
219
“true autonomy in the cyber domain”: Michael Walker, interview, December 5, 2016.
219
comparable to a “competent” computer security professional: David Brumley, interview, November 24, 2016.
219
DEF CON hacking conference: Daniel Tkacik, “CMU Team Wins Fourth ‘World Series of Hacking’ Competition,” CMU.edu, July 31, 2017.
219
Brumley’s team from Carnegie Mellon: Ibid.
219
Mirai: Brian Krebs, “Who Makes the IoT Things Under Attack?” Krebs on Security, October 3, 2016, https://krebsonsecurity.com/2016/10/who-makes-the-iot-things-under-attack/.
219
massive DDoS attack: Brian Krebs, “KrebsOnSecurity Hit With Record DDoS,” Krebs on Security, September 21, 2016, https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/.
219
most IoT devices are “ridiculous vulnerable”: David Brumley, interview, November 24, 2016.
219
6.4 billion IoT devices: “Gartner Says 6.4 Billion Connected,” Gartner, November 10, 2015, http://www.gartner.com/newsroom/id/3165317.
220
“check all these locks”: David Brumley, interview, November 24, 2016.
220
“no difference” between the technology: Ibid.
220
“All computer security technologies are dual-use”: Michael Walker, interview, December 5, 2016.
220
“you have to trust the researchers”: David Brumley, interview, November 24, 2016.
220
“It’s going to take the same kind”: Michael Walker, interview, December 5, 2016.
221
“I’m not saying that we can change to a place”: Ibid.
221
“It’s scary to think of Russia”: David Brumley, interview, November 24, 2016.
221
“counter-autonomy”: David Brumley, “Winning Cyber Battles: The Next 20 Years,” unpublished working paper, November 2016.
221
“trying to find vulnerabilities”: David Brumley, interview, November 24, 2016.
221
“you play the opponent”: Ibid.
221
“It’s a little bit like a Trojan horse”: Ibid.
222
“computer equivalent to ‘the long con’”: Brumley, “Winning Cyber Battles: The Next 20 Years.”
222
“Make no mistake, cyber is a war”: Ibid.
222
F-35 . . . tens of millions of lines of code: Jacquelyn Schneider, “Digitally-Enabled Warfare: The Capability-Vulnerability Paradox,” Center for a New American Security, Washington DC, August 29, 2016, https://www.cnas.org/publications/reports/digitally-enabled-warfare-the-capability-vulnerability-paradox.
223
Hacking back is when: Dorothy E. Denning, “Framework and Principles for Active Cyber Defense,” December 2013, 3.
223
Hacking back can inevitably draw in third parties: Dan Goodin, “Millions of Dynamic DNS Users Suffer after Microsoft Seizes No-IP Domains,” Ars Technica, June 30, 2014, https://arstechnica.com/security/2014/06/millions-of-dymanic-dns-users-suffer-after-microsoft-seizes-no-ip-domains/.
223
Hacking back is controversial: Hannah Kuchler, “Cyber Insecurity: Hacking Back,” Financial Times, July 27, 2015, https://www.ft.com/content/c75a0196-2ed6-11e5-8873-775ba7c2ea3d.
223
“Every action accelerates”: Steve Rosenbush, “Cyber Experts Draw Line Between Active Defense, Illegal Hacking Back,” Wall Street Journal, July 28, 2016, https://blogs.wsj.com/cio/2016/07/28/cyber-experts-draw-line-between-active-defense-illegal-hacking-back/.
223
Coreflood botnet: Denning, 6.
223
Automated hacking back would delegate: “Hacking Back: Exploring a New Option of Cyber Defense,” InfoSec Resources, November 8, 2016, http://resources.infosecinstitute.com/hacking-back-exploring-a-new-option-of-cyber-defense/.
223
“autonomous cyber weapons could automatically escalate”: Patrick Lin, remarks at conference, “Cyber Weapons and Autonomous Weapons: Potential Overlap, Interaction and Vulnerabilities,” UN Institute for Disarmament Research, New York, October 9, 2015, http://www.unidir.org/programmes/emerging-security-threats/the-weaponization-of-increasingly-autonomous-technologies-addressing-competing-narratives-phase-ii/cyber-weapons-and-autonomous-weapons-potential-overlap-interaction-and-vulnerabilities. Comment at 5:10.
224
Automated hacking back is a theoretical concept: Alexander Velez-Green, “When ‘Killer Robots’ Declare War,” Defense One, April 12, 2015, http://www.defenseone.com/ideas/2015/04/when-killer-robots-declare-war/109882/.
224
automate “spear phishing” attacks: Karen Epper Hoffman, “Machine Learning Can Be Used Offensively to Automate Spear Phishing,” Infosecurity Magazine, August 5, 2016, https://www.infosecurity-magazine.com/news/bhusa-researchers-present-phishing/.
224
automatically develop “humanlike” tweets: John Seymour and Philip Tully, “Weaponizing data science for social engineering: Automated E2E spear phishing on Twitter,” https://www.blackhat.com/docs/us-16/materials/us-16-Seymour-Tully-Weaponizing-Data-Science-For-Social-Engineering-Automated-E2E-Spear-Phishing-On-Twitter-wp.pdf.
224
“in offensive cyberwarfare”: Eric Messinger, “Is It Possible to Ban Autonomous Weapons in Cyberwar?,” Just Security, January 15, 2015, https://www.justsecurity.org/19119/ban-autonomous-weapons-cyberwar/.
225
estimated 8 to 15 million computers worldwide: “Virus Strikes 15 Million PCs,” UPI, January 26, 2009, http://www.upi.com/Top_News/2009/01/26/Virus-strikes-15-million-PCs/19421232924206/.
225
method to counter Conficker: “Clock ticking on worm attack code,
” BBC News, January 20, 2009, http://news.bbc.co.uk/2/hi/technology/7832652.stm.
225
brought Conficker to heel: Microsoft Security Intelligence Report: Volume 11 (11), Microsoft, 2011.
226
“prevent and react to countermeasures”: Alessandro Guarino, “Autonomous Intelligent Agents in Cyber Offence,” in K. Podins, J. Stinissen, M. Maybaum, eds., 2013 5th International Conference on Cyber Conflict (Tallinn, Estonia: NATO CCD COE Publications, 2013), https://ccdcoe.org/cycon/2013/proceedings/d1r1s9_guarino.pdf.
226
“the synthesis of new logic”: Michael Walker, interview, December 5, 2016.
226
“those are a possibility and are worrisome”: David Brumley, interview, November 24, 2016.
227
“Defense is powered by openness”: Michael Walker, interview, December 5, 2016.
227
“I tend to view everything as a system”: David Brumley, interview, November 24, 2016.
227
what constitutes a “cyberweapon”: Thomas Rid and Peter McBurney, “Cyber-Weapons,” The RUSI Journal, 157 (2012):1, 6–13.
227
specifically exempts cyberweapons: Department of Defense, “Department of Defense Directive Number 3000.09, 2.
228
“goal is not offense”: Bradford Tousley, interview, April 27, 2016.
228
“the narrow cases where we will allow”: Bob Work, interview, June 22, 2016.
228
“We’ll work it through”: Ibid.
230
“they would just shut it down”: Ibid.
15 “Summoning the Demon”: The Rise of Intelligent Machines
231
cannot piece these objects together: Machines have been able to caption images with reasonable accuracy, describing in a general sense what the scene depicts. For an overview of current abilities and limitations in scene interpretation, see JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,” 10.
232
Brain imaging: “Human Connectome Project | Mapping the Human Brain Connectivity,” accessed June 15, 2017, http://www.humanconnectomeproject.org/. “Meet the World’s Most Advanced Brain Scanner,” Discover Magazine, accessed June 15, 2017, http://discovermagazine.com/2013/june/08-meet-the-worlds-most-advanced-brain-scanner.
232