The Age of Surveillance Capitalism

by Shoshana Zuboff

  46. Max Friedrich Meyer, Psychology of the Other-One (Missouri Book Company, 1921), 146, http://archive.org/details/cu31924031214442.

  47. Meyer, Psychology, 147.

  48. Meyer, 402, 406.

  49. Meyer, 411–12, 420.

  50. Meyer, 402.

  51. Meyer, 404.

  52. B. F. Skinner, The Behavior of Organisms: An Experimental Analysis (Acton, MA: Copley, 1991), 3.

  53. Skinner, The Behavior of Organisms, 4–6.

  54. B. F. Skinner, Science and Human Behavior, (Kindle Edition: Free Press, 2012), 228–29.

  55. B. F. Skinner, Beyond Freedom & Dignity (Indianapolis: Hackett, 2002), 163.

  56. Skinner, Beyond Freedom & Dignity, 19–20.

  57. Skinner, 21, 44, 58.

  58. Skinner, Science and Human Behavior, 20.

  59. Skinner, Beyond Freedom & Dignity, 4–5.

  60. Skinner, 5–6 (italics mine).

  61. Skinner, 59.

  62. “Gambling Is a Feature of Capitalism—Not a Bug,” Prospect, April 2017, http://www.prospectmagazine.co.uk/magazine/gambling-is-a-feature-of-capitalism-not-a-bug; Natasha Dow Schüll, Addiction by Design: Machine Gambling in Las Vegas (Princeton, NJ: Princeton University Press, 2014); Howard J. Shaffer, “Internet Gambling & Addiction,” Harvard Medical School: Division on Addictions, Cambridge Health Alliance, January 16, 2004; Michael Kaplan, “How Vegas Security Drives Surveillance Tech Everywhere,” Popular Mechanics, January 1, 2010, http://www.popularmechanics.com/technology/how-to/computer-security/4341499; Adam Tanner, What Stays in Vegas: The World of Personal Data—Lifeblood of Big Business—and the End of Privacy as We Know It (New York: PublicAffairs, 2014); Chris Nodder, Evil by Design: Interaction Design to Lead Us into Temptation (Indianapolis: Wiley, 2013); Julian Morgans, “Your Addiction to Social Media Is No Accident,” Vice, May 18, 2017, https://www.vice.com/en_us/article/vv5jkb/the-secret-ways-social-media-is-built-for-addiction; “Reasons for Playing Slot Machines Rather Than Table Games in the U.S.,” Statista, 2017, https://www.statista.com/statistics/188761/reasons-for-playing-slot-machines-more-than-table-games-in-the-us.

  63. Skinner, Science and Human Behavior, 105–6, 282 (italics mine). For insight into behavior technologies in casinos, see Tanner, What Stays in Vegas.

  64. Skinner, Science and Human Behavior, 105–6.

  65. Skinner, 21.

  66. Skinner, 282.

  67. Skinner, 282.

  68. Some may be surprised that George Orwell characterized 1984 to his friend Anthony Powell as “a utopia written as a novel.” See Robert McCrum, “1984: The Masterpiece That Killed George Orwell,” Guardian, May 9, 2009, http://www.theguardian.com/books/2009/may/10/1984-george-orwell. B. F. Skinner characterized Walden Two as “a novel about a Utopian community.” B. F. Skinner, Walden Two (Indianapolis: Hackett, 2005), vi.

  69. See Skinner’s foreword to the 1976 edition of Walden Two: B. F. Skinner, Walden Two (New York: Macmillan, 1976).

  70. George Orwell, 1984 (Boston: Houghton Mifflin Harcourt, 2017), 548, 551.

  71. Orwell, 1984, 637–38.

  72. Skinner, Walden Two, 275–76.

  73. Skinner, 137, 149.

  CHAPTER THIRTEEN

  1. B. F. Skinner, Walden Two (Indianapolis: Hackett, 2005), 242–43.

  2. B. F. Skinner, “To Know the Future,” Behavior Analyst 13, no. 2 (1990): 104.

  3. Skinner, “To Know the Future,” 106.

  4. Hannah Arendt, Essays in Understanding (New York: Schocken, 1994), 319.

  5. Hannah Arendt, The Human Condition (Chicago: University of Chicago Press, 1998), 322.

  6. Hannah Arendt, The Origins of Totalitarianism (New York: Schocken, 2004), 620.

  7. “Trust,” Our World in Data, August 3, 2017, https://ourworldindata.org/trust.

  8. “Public Trust in Government: 1958–2017,” Pew Research Center for the People and the Press, May 3, 2017, http://www.people-press.org/2017/05/03/public-trust-in-government-1958-2017.

  9. Peter P. Swire, “Privacy and Information Sharing in the War on Terrorism,” Villanova Law Review 51, no. 4 (2006): 951. See also Kristen E. Eichensehr, “Public-Private Cybersecurity,” Texas Law Review 95, no. 3 (2017), https://texaslawreview.org/public-private-cybersecurity.

  10. Joseph Menn, “Facebook, Twitter, Google Quietly Step Up Fight Against Terrorist Propaganda,” Sydney Morning Herald, December 7, 2015, http://www.smh.com.au/technology/technology-news/facebook-twitter-google-quietly-step-up-fight-against-terrorist-propaganda-20151206-glgvj2.html.

  11. Menn, “Facebook, Twitter, Google.” See also Jim Kerstetter, “Daily Report: Tech Companies Pressured on Terrorist Content,” Bits Blog, December 8, 2015, https://bits.blogs.nytimes.com/2015/12/08/daily-report-tech-companies-pressured-on-terrorist-content; Mark Hosenbell and Patricia Zengerle, “Social Media Terrorist Activity Bill Returning to Senate,” Reuters, December 7, 2015, http://www.reuters.com/article/us-usa-congress-socialmedia-idUSKBN0TQ 2E520151207.

  12. Dave Lee, “‘Spell-Check for Hate’ Needed, Says Google’s Schmidt,” BBC News, December 7, 2015, http://www.bbc.com/news/technology-35035087.

  13. Danny Yadron, “Agenda for White House Summit with Silicon Valley,” Guardian, January 7, 2016, https://www.theguardian.com/technology/2016/jan/07/white-house-summit-silicon-valley-tech-summit-agenda-terrorism; Danny Yadron, “Revealed: White House Seeks to Enlist Silicon Valley to ‘Disrupt Radicalization,’” Guardian, January 8, 2016, http://www.theguardian.com/technology/2016/jan/07/white-house-social-media-terrorism-meeting-facebook-apple-youtube.

  14. Kashmir Hill, “The Government Wants Silicon Valley to Build Terrorist-Spotting Algorithms. But Is It Possible?” Fusion, January 14, 2016, http://fusion.net/story/255180/terrorist-spotting-algorithm.

  15. Stefan Wagstyl, “Germany to Tighten Security in Wake of Berlin Terror Attack,” Financial Times, January 11, 2017, https://www.ft.com/content/bf7972f4-d759-11e6-944b-e7eb37a6aa8e.

  16. John Mannes, “Facebook, Microsoft, YouTube and Twitter Form Global Internet Forum to Counter Terrorism,” TechCrunch, June 26, 2017, http://social.techcrunch.com/2017/06/26/facebook-microsoft-youtube-and-twitter-form-global-internet-forum-to-counter-terrorism; “Partnering to Help Curb the Spread of Terrorist Content Online,” Google, December 5, 2016, http://www.blog.google:443/topics/google-europe/partnering-help-curb-spread-terrorist-content-online.

  17. “Five Country Ministerial 2017: Joint Communiqué,” June 28, 2017, https://www.publicsafety.gc.ca/cnt/rsrcs/pblctns/fv-cntry-mnstrl-2017/index-en.aspx.

  18. “European Council Conclusions on Security and Defence, 22/06/2017,” June 22, 2017, http://www.consilium.europa.eu/en/press/press-releases/2017/06/22/euco-security-defence.

  19. “G20 Leaders’ Statement on Countering Terrorism—European Commission Press Release,” July 7, 2017, http://europa.eu/rapid/press-release_STATEMENT-17-1955_en.htm; Jamie Bartlett, “Terrorism Adds the Backdrop to the Fight for Internet Control,” Financial Times, June 6, 2017, https://www.ft.com/content/e47782fa-4ac0-11e7-919a-1e14ce4af89b.

  20. Spencer Ackerman and Sam Thielman, “US Intelligence Chief: We Might Use the Internet of Things to Spy on You,” Guardian, February 9, 2016, http://www.theguardian.com/technology/2016/feb/09/internet-of-things-smart-home-devices-government-surveillance-james-clapper.

  21. Matt Olsen, Bruce Schneier, and Jonathan Zittrain, “Don’t Panic: Making Progress on the ‘Going Dark’ Debate,” Berkman Klein Center for Internet & Society at Harvard, February 1, 2016, 13.

  22. Haley Sweetland Edwards, “Alexa Takes the Stand,” Time, May 15, 2017; Tom Dotan and Reed Albergotti, “Amazon Echo and the Hot Tub Murder,” Information, December 27, 2016, https://www.theinformation.com/amazon-echo-and-the-hot-tub-murder.

  23. Parmy Olson, “Fitbit Data Now Being Used in the Courtroom,” Forbes, November 16, 2014, http://www.forbes.com/sites/parmyolson/2014/11/16/fitbit-data-court-room-personal-injury-claim; Kate Crawford, “When Fitbit Is the Expert Witness,” Atlantic, November 19, 2014, http://
www.theatlantic.com/technology/archive/2014/11/when-fitbit-is-the-expert-witness/382936; Ms. Smith, “Cops Use Pacemaker Data to Charge Man with Arson, Insurance Fraud,” CSO, January 30, 2017, http://www.csonline.com/article/3162740/security/cops-use-pacemaker-data-as-evidence-to-charge-homeowner-with-arson-insurance-fraud.html.

  24. Jonah Engel Bromwich, Mike Isaac, and Daniel Victor, “Police Use Surveillance Tool to Scan Social Media, A.C.L.U. Says,” New York Times, October 11, 2016, http://www.nytimes.com/2016/10/12/technology/aclu-facebook-twitter-instagram-geofeedia.html.

  25. Jennifer Levitz and Zusha Elinson, “Boston Plan to Track Web Draws Fire,” Wall Street Journal, December 5, 2016.

  26. Lee Fang, “The CIA Is Investing in Firms That Mine Your Tweets and Instagram Photos,” Intercept, April 14, 2016, https://theintercept.com/2016/04/14/in-undisclosed-cia-investments-social-media-mining-looms-large.

  27. Ashley Vance and Brad Stone, “Palantir, the War on Terror’s Secret Weapon,” Bloomberg.com, September 22, 2011, http://www.bloomberg.com/news/articles/2011-11-22/palantir-the-war-on-terrors-secret-weapon; Ali Winston, “Palantir Has Secretly Been Using New Orleans to Test Its Predictive Policing Technology,” Verge, February 27, 2018.

  28. Rogier Creemers, “China’s Chilling Plan to Use Social Credit Ratings to Keep Score on Its Citizens,” CNN.com, October 27, 2015, https://www.cnn.com/2015/10/27/opinions/china-social-credit-score-creemers/index.html.

  29. Mara Hvistendahl, “Inside China’s Vast New Experiment in Social Ranking,” Wired, December 14, 2017, https://www.wired.com/story/age-of-social-credit.

  30. Hvistendahl, “Inside China’s Vast New Experiment.”

  31. Amy Hawkins, “Chinese Citizens Want the Government to Rank Them,” Foreign Policy, May 24, 2017, https://foreignpolicy.com/2017/05/24/chinese-citizens-want-the-government-to-rank-them.

  32. Zhixin Feng et al., “Social Trust, Interpersonal Trust and Self-Rated Health in China: A Multi-level Study,” International Journal for Equity in Health 15 (November 8, 2016), https://doi.org/10.1186/s12939-016-0469-7.

  33. Arthur Kleinman et al., Deep China: The Moral Life of the Person (Berkeley: University of California Press, 2011); Mette Halskov Hansen, iChina: The Rise of the Individual in Modern Chinese Society, ed. Rune Svarverud (Copenhagen: Nordic Institute of Asian Studies, 2010); Yunxiang Yan, The Individualization of Chinese Society (Oxford: Bloomsbury Academic, 2009).

  34. Hawkins, “Chinese Citizens Want the Government to Rank Them.”

  35. Hvistendahl, “Inside China’s Vast New Experiment.”

  36. Hvistendahl.

  37. Masha Borak, “China’s Social Credit System: AI-Driven Panopticon or Fragmented Foundation for a Sincerity Culture?” TechNode, August 23, 2017, https://technode.com/2017/08/23/chinas-social-credit-system-ai-driven-panop ticon-or-fragmented-foundation-for-a-sincerity-culture.

  38. “China Invents the Digital Totalitarian State,” Economist, December 17, 2016, http://www.economist.com/news/briefing/21711902-worrying-implications-its-social-credit-project-china-invents-digital-totalitarian.

  39. Shi Xiaofeng and Cao Yin, “Court Blacklist Prevents Millions from Flying, Taking High-Speed Trains,” Chinadaily.com, February 14, 2017, http://www.chinadaily.com.cn/china/2017-02/14/content_28195359.htm.

  40. “China Moving Toward Fully Developed Credit Systems,” Global Times, June 6, 2017, http://www.globaltimes.cn/content/1052634.shtml.

  41. Hvistendahl, “Inside China’s Vast New Experiment.”

  42. Yaxing Yao, Davide Lo Re, and Yang Wang, “Folk Models of Online Behavioral Advertising,” in Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’17 (New York: ACM, 2017), 1957–69, https://doi.org/10.1145/2998181.2998316.

  43. “China Invents the Digital Totalitarian State.”

  44. Christopher Lunt, United States Patent: 9100400—Authorization and authentication based on an individual’s social network, issued August 4, 2015, http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d =PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1 =9100400.PN.&OS=PN/9100400&RS=PN/9100400. The critical paragraph in the patent reads as follows: “In a fourth embodiment of the invention, the service provider is a lender. When an individual applies for a loan, the lender examines the credit ratings of members of the individual’s social network who are connected to the individual through authorized nodes. If the average credit rating of these members is at least a minimum credit score, the lender continues to process the loan application. Otherwise, the loan application is rejected.”

  45. Christer Holloman, “Your Facebook Updates Now Determine Your Credit Score,” Guardian, August 28, 2014, http://www.theguardian.com/media-net work/media-network-blog/2014/aug/28/social-media-facebook-credit-score-banks; Telis Demos and Deepa Seetharaman, “Facebook Isn’t So Good at Judging Your Credit After All,” Wall Street Journal, February 24, 2016, http://www.wsj.com/articles/lenders-drop-plans-to-judge-you-by-your-facebook-friends-1456309801; Yanhao Wei et al., “Credit Scoring with Social Network Data” (SSRN Scholarly Paper, Rochester, NY: Social Science Research Network, July 1, 2014), https://papers.ssrn.com/abstract=2475265; Daniel Bjorkegren and Darrell Grissen, “Behavior Revealed in Mobile Phone Usage Predicts Loan Repayment” (SSRN Scholarly Paper, Rochester, NY: Social Science Research Network, July 13, 2015), https://papers.ssrn.com/abstract=2611775.

  46. Creemers, “China’s Chilling Plan.”

  47. See Dan Strumpf and Wenxin Fan, “Who Wants to Supply China’s Surveillance State? The West,” Wall Street Journal, November 1, 2017, https://www.wsj.com/articles/who-wants-to-supply-chinas-surveillance-state-the-west-1509540111.

  48. Carl J. Friedrich, “The Problem of Totalitarianism—an Introduction,” in Totalitarianism, ed. Carl J. Friedrich (New York: Grosset & Dunlap, 1964), 1–2.

  CHAPTER FOURTEEN

  1. Mark Weiser, “The Computer for the 21st Century,” Scientific American, July 1999, 89.

  2. Satya Nadella, “Build 2017,” Build Conference 2017, Seattle, May 10, 2017, https://ncmedia.azureedge.net/ncmedia/2017/05/Build-2017-Satya-Nadella-transcript.pdf.

  3. Eric Schmidt, “Alphabet’s Eric Schmidt: We Should Embrace Machine Learning—Not Fear It,” Newsweek, January 10, 2017, http://www.newsweek.com/2017/01/20/google-eric-schmidt-embrace-machine-learning-not-fear-it-540369.html.

  4. Richard Waters, “FT Interview with Google Co-founder and CEO Larry Page,” Financial Times, October 31, 2014, http://www.ft.com/intl/cms/s/2/3173f19e-5fbc-11e4-8c27-00144feabdc0.html#axzz3JjXPNno5.

  5. Marcus Wohlsen, “Larry Page Lays Out His Plan for Your Future,” Wired, March 2014, https://www.wired.com/2014/03/larry-page-using-google-build-future-well-living.

  6. Waters, “FT Interview with Google Co-founder”; Vinod Khosla, “Fireside Chat with Google Co-founders, Larry Page and Sergey Brin,” Khosla Ventures, July 3, 2014, http://www.khoslaventures.com/fireside-chat-with-google-co-founders-larry-page-and-sergey-brin.

  7. Miguel Helft, “Fortune Exclusive: Larry Page on Google,” Fortune, December 11, 2012, http://fortune.com/2012/12/11/fortune-exclusive-larry-page-on-google.

  8. Khosla, “Fireside Chat.”

  9. Larry Page, “2013 Google I/O Keynote,” Google I/O, May 15, 2013, http://www.pcworld.com/article/2038841/hello-larry-googles-page-on-negativity-laws-and-competitors.html.

  10. “Facebook’s (FB) CEO Mark Zuckerberg on Q4 2014 Results—Earnings Call Transcript,” Seeking Alpha, January 29, 2015, https://seekingalpha.com/article/2860966-facebooks-fb-ceo-mark-zuckerberg-on-q4-2014-results-earnings-call-transcript.

  11. See Ashlee Vance, “Facebook: The Making of 1 Billion Users,” Bloomberg.com, October 4, 2012, http://www.bloomberg.com/news/articles/2012-10-04/face book-the-making-of-1-billion-users.

  12. “Facebook’s (FB) CEO Mark Zuckerberg on Q4 2014 Results.”

  13. “Facebook (FB) Mark Elliot Zuckerberg on Q1 2016 Results—Earnings Call Transcript,” Seeking Alpha, April 28, 2016, https://seekinga
lpha.com/article/3968783-facebook-fb-mark-elliot-zuckerberg-q1-2016-results-earnings-call-transcript.

  14. Mark Zuckerberg, “Building Global Community,” Facebook, February 16, 2017, https://www.facebook.com/notes/mark-zuckerberg/building-global-community/10154544292806634.

  15. Mark Zuckerberg, “Facebook CEO Mark Zuckerberg’s Keynote at F8 2017 Conference (Full Transcript),” April 19, 2017, https://singjupost.com/facebook-ceo-mark-zuckerbergs-keynote-at-f8-2017-conference-full-transcript.

  16. Johann Wolfgang von Goethe, “The Sorcerer’s Apprentice,” German Stories at Virginia Commonwealth University, 1797, http://germanstories.vcu.edu/goethe/zauber_e4.html.

  17. Frank E. Manuel and Fritzie P. Manuel, Utopian Thought in the Western World (Cambridge, MA: Belknap Press, 1979), 20.

  18. Manuel and Manuel, Utopian Thought, 23 (italics mine).

  19. Todd Bishop and Nat Levy, “With $256 Billion, Apple Has More Cash Than Amazon, Microsoft and Google Combined,” GeekWire, May 2, 2017, https://www.geekwire.com/2017/256-billion-apple-cash-amazon-microsoft-google-combined.

  20. Manuel and Manuel, Utopian Thought in the Western World, 9.

  21. Zuckerberg, “Building Global Community.”

  22. “Facebook CEO Mark Zuckerberg’s Keynote at F8 2017 Conference.”

  23. Nadella, “Build 2017.”

  24. Satya Nadella, Chen Qiufan, and Ken Liu, “The Partnership of the Future,” Slate, June 28, 2016, http://www.slate.com/articles/technology/future_tense/2016/06/microsoft_ceo_satya_nadella_humans_and_a_i_can_work_together_to_solve_society.html.

  25. Nadella, “Build 2017.”

  26. Nadella.

  27. Nadella (italics mine).

  28. Nadella (italics mine).

  29. Nadella.