Overcomplicated

by Samuel Arbesman

  “When Technology Ceases to Amaze” by Robert Herritt in The New Atlantis 41, Winter 2014, pages 121–31, is a great essay about technological wonder, complexity, and amazement.

  “In the Beginning . . . Was the Command Line” by Neal Stephenson, an essay and also a short book, is essentially a long and winding meditation on computing. Though outdated, it contains a great deal of wisdom on connection to and detachment from technology.

  “I, Pencil” by Leonard E. Read is a brief essay that explores the highly interconnected socioeconomic system involved in manufacturing a pencil, the totality of which no single individual understands.

  I also recommend looking at the writings of Edsger Dijkstra in the E. W. Dijkstra Archive, many of which are classics in computer science and border on philosophical musings on the nature of technology. Available online at https://www.cs.utexas.edu/~EWD/.

  For additional reading, please go to arbesman.net to find essays of mine on these topics.

  Acknowledgments

  First of all, thank you to my editor, Niki Papadopoulos, for the invaluable support throughout the entire book-writing process. Max Brockman, thanks for helping me shape the idea for this book.

  Portions of this book have been developed elsewhere, as well as appearing in modified form in numerous places, including Aeon, Nautilus, Slate, Wired Opinion, Edge.org’s “The Edge Question—2015: What Do You Think About Machines That Think?” and Arc. Thank you to these outlets for providing me with an opportunity to work out my ideas in public. I’d also like to thank Wired Science for having let me develop many of the ideas in this book on my blog there, where they appeared in a considerably more incoherent form. Also, those of you who subscribe to my email newsletter were able to experience early versions of some of these topics as well.

  Thank you to all my early readers and interlocutors for your generosity of time and wealth of expertise, both of which are quite rare in this modern era. These include Josh Arbesman, Zev Berger, Andrew Blum, Aaron Clauset, Lori Emerson, Joshua Fairfield, Henry Farrell, Laurence Gonzales, Edward Jung, Aaron Kahn, Dan Katz, Mykel Kochenderfer, Steven Miller, Megan Owen, Elnatan Reisner, Nahum Shalman, Jacob Sherman, Ted Steinberg, Brian Stephens, Harry Surden, and Jevin West. To all of you, and anyone else who helped shape the ideas found here, many thanks for helping make this book better, and preventing some pretty ridiculous goofs. That being said, all errors remain my own. I’d also like to single out in particular the philosophy research group at the University of Kansas that examines human understanding and software. As a visiting scholar in the philosophy department at KU, I am a part of this small group, and all the feedback and ideas from the members of this group—John Symons, Jack Horner, Ramon Alvarado—have improved my thinking on the topics of this book immeasurably. David Steen generously helped guide my thinking about wildlife ecology and field biology. Josh Sunshine provided a great deal of support and advice, fielding an innumerable number of questions related to computer science and software engineering. Michael Vitevitch was invaluable with his expertise and guidance on the nature of language and how we process it. Michael Barr and Philip Koopman also provided useful advice on the problem of unintended acceleration and technological complexity in Toyotas.

  I am also indebted to the Ewing Marion Kauffman Foundation, which provided me with a great environment for writing much of this book. Thank you to the Foundation and my colleagues there for your incredible feedback and warmth, especially in letting me work in not too monk-like a fashion on this text.

  In addition, the team at Lux Capital has been enormously supportive of this project, and provided numerous insights and feedback. Thanks so much to you all.

  Finally, my family. My parents provided much-needed support and advice, as well as detailed feedback on many drafts, and my grandfather is the best sounding board I know. I am incredibly grateful to you all. To my wife, Debra, you gave me the space and time to write this book and make sure that it was the best version possible. Your patience during this process, from reading early drafts to hearing me rant about technology, has been astonishing and wonderful.

  And Abigail: you love books, though currently you are enamored of those that are considerably shorter and often have sturdier pages than this one. Your ability to bring joy to me every day is something that I still find incredible. Your support throughout this book from its inception—even though it consisted primarily of hugs, cuddles, giggles, and delightful chatter—is more than I ever could have hoped for. While I hope this book finds a broad audience, ultimately it’s written for you and your new brother, Nathan. You both will come of age in a world that I can only begin to imagine, one far different from the one we live in today and far less understandable. This book hopes to explain, in a small way, why you shouldn’t be frightened by this future.

  Notes

  INTRODUCTION

  a lot of buggy software: Some speculate that the WSJ.com outage was caused by an overload when subscribers thronged to read about the NYSE shutdown. Jose Pagliery, “Tech Fail! Explaining Today’s 3 Big Computer Errors,” CNN Money, July 8, 2015, http://money.cnn.com/2015/07/08/technology/united-nyse-wsj-down/. For more on this glitch and what it portends, see Zeynep Tufekci, “Why the Great Glitch of July 8th Should Scare You,” The Message, July 8, 2015, https://medium.com/message/why-the-great-glitch-of-july-8th-should-scare-you-b791002fff03#.cd6hchnur.

  As one security expert stated: Andrea Chang and Tracey Lien, “Outages at NYSE, United Airlines, WSJ.com Expose Digital Vulnerabilities,” July 8, 2015, http://www.latimes.com/business/technology/la-fi-tn-technical-problems-united-nyse-20150708-story.html.

  the particle accelerator in Strasbourg: Thomas Homer-Dixon, The Ingenuity Gap: Facing the Economic, Environmental, and Other Challenges of an Increasingly Complex and Unpredictable Future (New York: Alfred A. Knopf, 2000; repr. Vintage, 2002), 171.

  radical novelty: Edsger W. Dijkstra, “On the Cruelty of Really Teaching Computing Science,” E. W. Dijkstra Archive: The manuscripts of Edsger W. Dijkstra, 1930–2002, document no. EWD1036, December 1988, http://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1036.PDF [hand-printed original], http://www.cs.utexas.edu/users/EWD/transcriptions/EWD10xx/EWD1036.html [typed transcript]. Dijkstra made a similar point in his Association for Computing Machinery (ACM) Turing Lecture in 1972. E. W. Dijkstra, “The Humble Programmer,” Communications of the ACM 15, no. 10 (1972): 859–86. Not every technology is computational, but Dijkstra’s insight does impinge on much of our technological life.

  the Anthropocene, the Epoch of Humanity: For further reading, see Lee Billings, “Brave New Epoch,” Nautilus 009: January 30, 2014.

  a journal article in Scientific Reports: Neil Johnson et al., “Abrupt Rise of New Machine Ecology Beyond Human Response Time,” Scientific Reports 3:2627, September 11, 2013.

  with humans on the sidelines: “Back in 2008, when it first occurred to Brad Katsuyama that the stock market had become a black box whose inner workings eluded ordinary human understanding . . .” Michael Lewis, “The Wolf Hunters of Wall Street,” The New York Times Magazine, March 31, 2014, http://www.nytimes.com/2014/04/06/magazine/flash-boys-michael-lewis.html.

  This phenomenon of “algorithm aversion”: Berkeley J. Dietvorst et al., “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err,” Journal of Experimental Psychology: General 144, no. 1 (2015), 114–26.

  When we delight at Google’s brain: As an example of this kind of reverence, a journalist wrote of feeling “almost levitated” when seeing a Google data center. Stephen Levy, “Google Throws Open Doors to Its Top-Secret Data Center,” Wired, October 17, 2012, http://www.wired.com/2012/10/ff-inside-google-data-center/. Another writer described the awe that can be induced by Facebook’s news feed algorithm: “The news feed algorithm’s outsize influence has given rise to a strand of criticism that treats it as if it possessed a mind of its own—as if it were some runic form of intelligence, loose
d on the world to pursue ends beyond the ken of human understanding.” Will Oremus, “Who Controls Your Facebook Feed,” Slate, January 3, 2016, http://www.slate.stfi.re/articles/technology/cover_story/2016/01/how_facebook_s_news_feed_algorithm_works.html.

  because of its creation by some perfect, infinite mind: “The worship of the algorithm” is discussed further in Ian Bogost, “The Cathedral of Computation,” The Atlantic, January 15, 2015, http://www.theatlantic.com/technology/archive/2015/01/the-cathedral-of-computation/384300/.

  CHAPTER 1: WELCOME TO THE ENTANGLEMENT

  the Challenger disaster: James Gleick, “Richard Feynman Dead at 69; Leading Theoretical Physicist,” The New York Times, February 17, 1988, http://www.nytimes.com/books/97/09/21/reviews/feynman-obit.html.

  car began accelerating uncontrollably: For further information on “unintended acceleration” in Toyota vehicles, see Ken Bensinger and Jerry Hirsch, “Jury Hits Toyota with $3-million Verdict in Sudden Acceleration Death Case,” Los Angeles Times, October 24, 2013, http://articles.latimes.com/2013/oct/24/autos/la-fi-hy-toyota-sudden-acceleration-verdict-20131024; Ralph Vartabedian and Ken Bensinger, “Runaway Toyota Cases Ignored,” Los Angeles Times, November 8, 2009, http://www.latimes.com/local/la-fi-toyota-recall8-2009nov08-story.html#page=1; Margaret Cronin Fisk, “Toyota Settles Oklahoma Acceleration Case After Verdict,” Bloomberg Business, October 25, 2013, http://www.bloomberg.com/news/articles/2013-10-25/toyota-settles-oklahoma-acceleration-case-after-jury-verdict; Associated Press, “Jury Finds Toyota Liable in Fatal Wreck in Oklahoma,” New York Times, October 25, 2013, http://www.nytimes.com/2013/10/25/business/jury-finds-toyota-liable-in-fatal-wreck-in-oklahoma.html.

  computer scientist Philip Koopman: Philip Koopman posted a talk with slides, “A Case Study of Toyota Unintended Acceleration and Software Safety,” on his blog Better Embedded System SW, October 3, 2014, http://betterembsw.blogspot.com/2014/09/a-case-study-of-toyota-unintended.html. See also the report by Michael Barr of the Barr Group on Bookout v. Toyota, http://www.safetyresearch.net/Library/BarrSlides_FINAL_SCRUBBED.pdf.

  in this case, unnecessarily complex: There is a distinction between inherent complexity and accidental complexity. The former is complexity that is required for a system to operate (including provision for various exceptions and special situations). The latter is closer to overcomplication, the complexity that often arises when a system grows by accretion and tinkering rather than a careful plan.

  an Ariane 5 rocket exploded: The Ariane rocket story is told in Homer-Dixon, The Ingenuity Gap. Homer-Dixon based part of his narrative on James Gleick, “A Bug and a Crash: Sometimes a Bug Is More Than a Nuisance,” 1996, http://www.around.com/ariane.html (which originally appeared in The New York Times Magazine, December 1996). For a similar discussion of proximate causes versus the underlying reasons for such sudden system failures, see Chris Clearfield and James Owen Weatherall, “Why the Flash Crash Really Matters,” Nautilus 023, April 23, 2015, http://nautil.us/issue/23/dominoes/why-the-flash-crash-really-matters.

  Three Mile Island nuclear disaster: Clearfield and Weatherall, “Why the Flash Crash Really Matters.”

  the system’s massive complexity: Essentially, the failure in each of these cases was due to endogenous complexity—the complexity that evolves within a large system—rather than just to any specific exogenous shock.

  popular narrative of the Challenger: It must be recognized that the Challenger accident was more complicated than the streamlined story we are often told about its cause. For example, engineers involved were aware of “the risk of catastrophic failure” of the space shuttle—though, as the following source notes, they could not pinpoint a specific reason—and objected to its launch. At the time, it seems, the engineers knew that “temperature might be a causal factor,” but were not certain of it. Wade Robison et al., “Representation and Misrepresentation: Tufte and the Morton Thiokol Engineers on the Challenger,” Science and Engineering Ethics 8, no. 1 (2002): 59–81, https://people.rit.edu/wlrgsh/FINRobison.pdf. Further details can also be found in this oral history of the accident, which indicates that engineers seem to have known the cause of the accident and that this information was given to Feynman to highlight in the hearing: Margaret Lazarus Dean, “An Oral History of the Space Shuttle Challenger Disaster,” Popular Mechanics, February, 2016, http://www.popularmechanics.com/space/a18616/an-oral-history-of-the-space-shuttle-challenger-disaster/.

  Whiggish view of progress: Philip Ball, “Science Fictions,” Aeon, October 29, 2012, http://aeon.co/magazine/science/philip-ball-history-science/.

  as the historian Ian Beacock writes: Ian Beacock, “Humanist among Machines,” Aeon, June 25, 2015, http://aeon.co/magazine/society/why-we-need-arnold-toynbees-muscular-humanism/.

  described by the sociologist Max Weber: Max Weber, “Science as a Vocation,” in From Max Weber: Essays in Sociology, trans. and ed. H. H. Gerth and C. Wright Mills, 129–56 (New York: Oxford University Press, 1946, repr. 1958), available online: http://anthropos-lab.net/wp/wp-content/uploads/2011/12/Weber-Science-as-a-Vocation.pdf.

  systems that can’t be grasped in their totality: This new way that our own creations confound us is echoed in a possibly apocryphal quote from Paul Valéry: “So the whole question comes down to this: can the human mind master what the human mind has made?” Quoted in Langdon Winner, Autonomous Technology: Technics-out-of-Control as a Theme in Political Thought (Cambridge, MA: The MIT Press, 1977), 13.

  “complicated” and “complex” systems: This is but one of likely very many distinctions between these two terms.

  Imagine water buoys: Thanks to Aaron Clauset for providing the example of tied-together buoys during a discussion.

  the infrastructure of our cities: In Kevin Kelly’s view, “Cities are technological artifacts, the largest technology we make.” What Technology Wants (New York: Viking, 2010), 81.

  could fill encyclopedias: David McCandless, “Codebases: Millions of Lines of Code,” infographic, v. 0.9, Information Is Beautiful, September 24, 2015, http://www.informationisbeautiful.net/visualizations/million-lines-of-code/. Assuming an encyclopedia has about 30,000 pages and each page could fit 1,000 lines of code, that means that by some estimates, one version of the Macintosh operating system could fill multiple encyclopedias.

  300,000 intersections with traffic signals: The number of “signalized intersections” in the United States is an estimate from the U.S. Department of Transportation’s Federal Highway Administration, http://mutcd.fhwa.dot.gov/knowledge/faqs/faq_part4.htm (last modified October 20, 2015).

  Autocorrect, which we often deride: Gideon Lewis-Kraus, “The Fasinatng . . . Frustrating . . . Fascinating History of Autocorrect,” Wired, July 22, 2014, http://www.wired.com/2014/07/history-of-autocorrect/.

  pages in the federal tax code: Laura Saunders, “Paper Trail,” sidebar to article “Don’t Make These Tax Mistakes: Fifteen Common Tax-Filing Errors That Can Cost You Dearly,” The Wall Street Journal, January 31, 2014.

  It exists across a rich spectrum: For a way to think about different levels of understanding of complex systems, see Stephen Jay Kline, Conceptual Foundations for Multidisciplinary Thinking (Stanford, CA: Stanford University Press, 1995), 268.

  Lee Felsenstein has told the story: Erik Sandberg-Diment, “A Computer Comes in from the Cold,” The New York Times, April 21, 1987, http://www.nytimes.com/1987/04/21/science/personal-computers-a-computer-comes-in-from-the-cold.html.

  computer scientist Gerard Holzmann: Gerard J. Holzmann, “Code Inflation,” IEEE Software (March/April 2015): 10–13.

  “too dense to be knowable”: Philip K. Howard, “Fixing Broken Government: Put Humans in Charge,” The Atlantic, September 22, 2014, http://www.theatlantic.com/politics/archive/2014/09/fixing-broken-government-put-humans-in-charge/380309/?single_page=true.

  the writer Quinn Norton has noted: Quinn Norton, “Everything is Broken,” The Message, May 20
, 2014, https://medium.com/message/81e5f33a24e1.

  Langdon Winner notes in his book: Winner, Autonomous Technology, 290–91.

  computer scientist Danny Hillis argues: Danny Hillis, “The Age of Digital Entanglement,” Scientific American, September 2010, 93.

  Take the so-called Flash Crash: Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, UK: Oxford University Press, 2014), 17. It is still not entirely clear, however, what caused the Flash Crash.

  Understanding something in a “good enough” way: See also César Hidalgo, Why Information Grows: The Evolution of Order, from Atoms to Economies (New York: Basic Books, 2015).

  CHAPTER 2: THE ORIGINS OF THE KLUGE

  the Internet first began to be developed: For more, see Barry M. Leiner et al., “Brief History of the Internet,” Internet Society, October 15, 2012, http://www.internetsociety.org/brief-history-internet.

  the source HTML of Google’s homepage: See Randall Munrose, “DNA,” xkcd, November 18, 2015, https://xkcd.com/1605/.

  Slate interactives editor Chris Kirk: Chris Kirk, “Battling My Daemons: My Email Made Me Miserable. So I Decided to Build My Own Email App from Scratch,” Slate, February 25, 2015, http://www.slate.com/articles/technology/technology/2015/02/email_overload_building_my_own_email_app_to_reach_inbox_zero.html.

  laws and regulations are technologies: The analogy between computer code and other kinds of codes—legal, moral—can be pushed quite far, but treating these as distinct types of systems is probably best.

  to establish the postal service: Constitution of the United States of America, Article I, Section 8: “To Establish Post Offices and Post Roads”; “Title 39—Postal Service,” Code of Federal Regulations (annual edition), revised July 1, 2003, available online: http://www.gpo.gov/fdsys/pkg/CFR-2003-title39-vol1/content-detail.html.