The Age of Surveillance Capitalism

by Shoshana Zuboff

  The TV’s “surveillance policy”—yes, even a TV has a surveillance policy now—reveals the layers of surveillance effort and commercial interest that operate outside of awareness in our homes. Samsung acknowledges that the voice commands aimed at triggering the TV’s voice-recognition capabilities are sent to a third party and adds, “Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party through your use of Voice Recognition.”23 Samsung disclaims responsibility for the policies of third-party firms, as nearly all surveillance policies do, including the one that actually collects and translates the talk of its unsuspecting customers. Samsung advises that “You should exercise caution and review the privacy statements applicable to the third-party websites and services you use.”24 The intrepid consumer determined to study these documents will find no succor in Nuance’s privacy policy, only the same catechism found in Samsung’s and that of nearly every company. It also encourages you to read the privacy policies of the companies to whom it’s selling your conversations, and so it goes: a forced march toward madness or surrender.25

  In California, at least, the legislature passed a law prohibiting connected TVs from collecting voice data without “prominently informing” customers and further outlawed the use of such data for third-party advertising.26 As we know from our examination of the dispossession cycle, however, the economic imperatives that drive surveillance capitalists to capture behavioral surplus are not easily deterred. By 2016, Samsung had doubled down on its smart-TV–based secret rendition and behavioral surplus supply chains, positioning its new models as the hub of a new “Samsung SmartThings smart-home ecosystem in an open platform that supports thousands of devices,” including fans, lights, thermostats, security cameras, and locks—and all with the aid of a single universal remote able to capture your every spoken command.27

  In 2017 the FTC reached a $2.2 million settlement of a complaint initiated by the Office of the New Jersey Attorney General against Vizio, one of the world’s largest manufacturers and distributors of internet-enabled smart televisions. Vizio’s supply operations appear to be even more aggressive than Samsung’s. Investigators discovered that “on a second-by-second basis, Vizio collected a selection of pixels on the screen that it matched to a database of TV, movie, and commercial content.” The company then identified additional viewing data “from cable or broadband service providers, set-top boxes, streaming devices, DVD players, and over-the-air broadcasts.” All of this amounted to as much as 100 billion data points each day just from the 11 million TVs identified in the settlement.28 Vizio disguised its supply operations behind a setting called “Smart Interactivity,” described to consumers as a feature that “enables program offers and suggestions” without any indication of its actual functions.

  In an unusually vivid blog post, the FTC describes Vizio’s direct sales of this behavioral surplus:

  Vizio then turned that mountain of data into cash by selling consumers’ viewing histories to advertisers and others. And let’s be clear: We’re not talking about summary information about national viewing trends. According to the complaint, Vizio got personal. The company provided consumers’ IP addresses to data aggregators, who then matched the address with an individual consumer or household. Vizio’s contracts with third parties prohibited the re-identification of consumers and households by name, but allowed a host of other personal details—for example, sex, age, income, marital status, household size, education, and home ownership. And Vizio permitted these companies to track and target its consumers across devices.29

  A concurring statement from FTC Acting Chairwoman Maureen K. Ohlhausen emphasized that the settlement broke new ground in its allegation that “individualized television viewing activity falls within the definition of sensitive information” that merits protection by the FTC.30 This finger in the dike would not hold back the tidal wave of similar incursions as the prediction imperative cracks the whip to drive the hunt for unexplored pieces of talk from daily life. Rendition takes command of even the most benign supply sources, such as toys, which have now become “toys that spy.” A new breed of interactive dolls and toy robots, including a girl doll called “My Friend Cayla,” turn out to be supply hubs for underage behavioral surplus, subjecting young children and their parents’ smartphones “to ongoing surveillance… without any meaningful data protection standards.”31

  The popular playthings, marketed by Genesis Toys, are bundled with a mobile application that, once downloaded to a smartphone, “provides the data processing” to enable the toy’s ability to capture and understand whatever the child says.32 Along the way, the app accesses most of the phone’s functions, including many that are irrelevant to the toy’s operations such as contact lists and the camera. The app establishes a Bluetooth connection that links the toy to the internet, and it records and uploads conversations as the toy actively engages the child in discussion. One doll targeted in the complaint systematically prompts children to submit a range of personal information, including where they live.

  The child’s conversations are translated into text by third-party voice-recognition software from, once again, Nuance Communications, and that information is used to retrieve answers to the child’s questions using Google Search and other web sources. Researchers discovered that the audio files of children’s talk (Nuance calls them “dialogue chunks”) are uploaded to the company’s servers, where they are analyzed and stored.33 As you might expect, those dialogue chunks continue their journey as behavioral surplus, in much the same way as Samsung’s TV-captured audio, to be sold again and again “for other services and products,” as the Genesis terms-of-service agreement indicates.

  Meanwhile, Mattel, one of the world’s largest toy companies, was gaining ground with its innovations in interactive, internet-enabled, machine-intelligence–powered toys, led by the new conversational Barbie Doll and its Barbie Dream House.34 The voice-activated smart dollhouse could respond to more than one hundred commands, such as “lower the elevator” and “turn on the disco ball,” a new kind of habituation exercise aimed at normalizing ubiquity in intimate spaces. “Barbie’s New Smart Home Is Crushing It So Hard,” exulted Wired. “Barbie’s ultimate crib is voice controlled.… It’s what a real smart home should be: Straight crushing it with universal voice control instead of a hodgepodge of disparate appliances hogging up app space on your phone.… The future is served.”35

  In this future, children learn the principles of the One Voice—a run time, a new interface. It is available everywhere to execute their commands, anticipate their desires, and shape their possibilities. The omnipresence of the One Voice, with its fractious, eager marketplace-of-you concealed under its skirts, changes many things. Intimacy as we have known it is compromised, if not eliminated. Solitude is deleted. The children will learn first that there are no boundaries between self and market. Later they will wonder how it could ever have been different.

  When Mattel hired a new CEO in January 2017, it’s no surprise that the company looked to Google, selecting its president for the Americas with responsibility for Google’s commercial and advertising sales operations.36 Most analysts agreed that the appointment heralded Mattel’s commitment to its innovations in internet-enabled toys and virtual reality, but the appointment underscores the shift in focus from making great products for you to collecting great data about you.

  The doll that was once a beloved mirror of a child’s unfettered imagination, along with all the other toys in the toy box—and the box, and the room that hosts the box, and the house that hosts the room—are all earmarked for rendition, calculation, connection, and profit. No longer mere things, they are reinvented as vehicles for a horde of commercial opportunities fabricated from our dialogue chunks and assorted gold dust.

  In 2017 Germany’s Federal Network Agency banned the Cayla doll as an illegal surveillance device and urged parents to destroy any such dolls
in their possession. In the US the FTC had yet to take any action against the doll or Genesis Toys. Meanwhile, the connected dollhouse prepares our children and families for the connected room (a project that Mattel announced in January 2017 and then nine months later shelved amid an uproar from parents and privacy advocates), which paves the way for the connected home, whose purveyors hope will numb us to the connected world as we travel the path of ubiquity’s manifest destiny and its promise of surveillance revenues.37

  In pursuit of the what and the how of voice surplus, the logic of competition is to corner as much supply as possible. The urge toward totality generates competitive pressures to become the run time and the new interface: the dominant, if not exclusive, medium through which we access and engage the apparatus as it engages us. It’s a race to corner all the talk as a prerequisite for achieving the privileged status of the One Voice, which will bestow upon the winner the ability to anticipate and monetize all the moments of all the people during all the days.

  The messianic urge toward totality and supremacy is evident in the rhetoric and strategies of key competitors in this race. Although Google, Microsoft, Amazon, and Samsung each have aspirations to dominate voice capture, it is Amazon, its machine learning assistant Alexa, and its expanding line of Echo hubs and Dot speakers that offer the most telling case here. Alexa appears to be a threshold event that will define Amazon not only as an aggressive capitalist but also as a surveillance capitalist.38

  Amazon aggressively opened Alexa to third-party developers in order to expand the assistant’s range of “skills,” such as reading a recipe or ordering a pizza. It also opened its platform to smart-home device makers from lighting systems to dishwashers, turning Alexa into a single voice for controlling your home systems and appliances. In 2015 Amazon announced that Alexa would be sold as a service, known as “Amazon Lex,” enabling any company to integrate Alexa’s brain into its products. Amazon Lex is described as “a service for building conversational interfaces into any application using voice and text.… Lex enables you to define entirely new categories of products.”39 As Alexa’s senior vice president explained, “Our goal is to try to create a kind of open, neutral ecosystem for Alexa… and make it as pervasive as we possibly can.”40

  By 2018, Amazon had inked deals with home builders, installing its Dot speakers directly into ceilings throughout the house as well as Echo devices and Alexa-powered door locks, light switches, security systems, door bells, and thermostats. As one report put it, “Amazon can acquire more comprehensive data on people’s living habits.…” The company wants to sell real-world services such as house cleaning, plumbing, and restaurant delivery, but according to some insiders, the vision is more far-reaching: an omniscient voice that knows all experience and anticipates all action.41 Already, forward-looking Amazon patents include the development of a “voice-sniffer algorithm” integrated into any device and able to respond to hot words such as “bought,” “dislike,” or “love” with product and service offers.42

  Amazon is on the hunt for behavioral surplus.43 This explains why the company joined Apple and Google in the contest for your car’s dashboard, forging alliances with Ford and BMW. “Shopping from the steering wheel” means hosting behavioral futures markets in your front seat. Alexa is ready with restaurant recommendations or advice on where to get your tires checked. “As pervasive as possible” explains why Amazon wants its Echo/Alexa device to also function as a home phone, able to make and receive calls; why it inked an agreement to install Echo in the nearly 5,000 rooms of the Wynn resort in Las Vegas; and why it is selling Alexa to call centers to automate the process of responding to live questions from customers by phone and text.44 Each expansion of Alexa’s territory increases the volume of voice surplus accrued to Amazon’s servers and fed to Alexa.

  The path to the coronation of the One Voice is not an easy one, and there are other travelers determined to elbow their way to the finish line. Google also wants its “personal assistant,” Google Home, to double as your home phone. Samsung reappears as another contender with its acquisition of “Viv,” a powerful new voice system designed by the original developers of Apple’s personal assistant Siri, who were frustrated with the constraints of Apple’s approach. Viv’s lead developer explained that “you can get things done by talking to things… a marketplace that will become the next big area.…”45

  If life is a wild horse, then the digital assistant is one more means by which that horse is to be broken by rendition. Unruly life is brought to heel, rendered as behavioral data and reimagined as a territory for browsing, searching, knowing, and modifying. Just as surveillance capitalism transformed the web into a market onslaught fueled by the capture and analysis of behavioral surplus, so everyday life is set to become a mere canvas for the explosion of a new always-on market cosmos dedicated to our behavior and from which there is no escape.

  II. Rendition of the Self

  “We are used to face-to-face interaction where words disappear.… I assumed that keyboard communication was like a letter or a phone call, but now I understand that it doesn’t disappear. The myth is that electronic communication is invisible.…”46 He was a brilliant research scientist at a large pharmaceuticals company that I called “Drug Corp” in my 1988 book, In the Age of the Smart Machine.47 I had spent several years visiting the research group there as they shifted much of their daily communications from face-to-face meetings to DIALOG, one of the world’s first “computer conferencing” systems. DIALOG was a precursor to a technology that we now call “social media.” The DIALOG platform created a new social space in which the scientific community at Drug Corp elaborated and extended “their networks of relationships, access to information, thoughtful dialogue, and social banter,” as I described it then. They embraced DIALOG with great enthusiasm, but it did not end well. “In time,” I wrote, “it became clear that they had also unwittingly exposed once evanescent and intangible aspects of their social exchange to an unprecedented degree of hierarchical scrutiny.” The interviews that stretched across those years documented the scientists’ gradual awakening to new dangers as dimensions of personal experience that were implicit and private suddenly became explicit and public in ways that they did not anticipate and with consequences that they lamented deeply.

  Thanks to the new computer-mediated milieu, the scientists’ social and professional conversations now appeared as an electronic text: visible, knowable, shareable. It enriched their work in many ways, but it also created unexpected vulnerabilities as dispositions, values, attitudes, and social interactions were recast as objects of scrutiny. In a series of conflicts that unfolded over the years, I watched as the managers and executives at Drug Corp were simply unable to quell their inclination to use the new social text as a means to evaluate, critique, and punish. On more than one occasion I learned of managers who had printed out pages of DIALOG conversations in order to spread them out on the floor and analyze opinions on a particular subject, sometimes actually taking scissors to the pages and organizing the entries by theme or person. In many cases these investigations were pure fact gathering, but in other instances managers wanted to identify the individuals who agreed with or opposed their directives.

  The enduring witness of the text was adapted as a medium through which managers attempted “to control and channel what had always been the most ephemeral aspects of subordinates’ behavior.”48 All the original excitement and promise melted into cynicism and anxiety as the scientists gradually withdrew from DIALOG, opting instead for a routine e-mail application and a preference for more-perfunctory, impersonal messages.

  Decades later, the children and grandchildren of those scientists, along with most of us, communicate freely through our smartphones and social media, unaware that we are reliving the bitter lessons of Drug Corp but now at an entirely new level of rendition. The scientists were rattled to see their casual talk reified and converted into an object of hierarchical inspection. Now, the interiors of our lives—si
mplistically summarized as “personality” or “emotions”—are recast as raw material available to anyone who can make or buy a new generation of supply-chain accessories and the means of production to analyze this new genus of behavioral surplus and fabricate extremely lucrative prediction products.

  “Personalization” is once again the euphemism that spearheads this generation of prediction products manufactured from the raw materials of the self. These innovations extend the logic of earlier iterations of dispossession: from web crawling to reality crawling to life crawling to self crawling. As has been the case in each iteration, insights and techniques once intended to illuminate and enrich quickly disappear into the magnetic field of the commercial surveillance project, only to reappear later as ever more cunning methods of supply, manufacture, and sales.

  The two billion and counting Facebook users are the most poignant descendants of Drug Corp’s scientists. Many of them joined Facebook to escape the pervasive hierarchical oversight of workplace communications that had become routine in the years since Drug Corp’s first experiments. Facebook, they once thought, was “our place,” as benign and taken for granted and as the old Ma Bell, a necessary utility for association, communication, and participation. Instead, Facebook became one of the most authoritative and threatening sources of predictive behavioral surplus from the depths. With a new generation of research tools it learned to plunder your “self” right through to your most intimate core. New supply operations can render as measurable behavior everything from the nuances of your personality to your sense of time, sexual orientation, intelligence, and scores of other personal characteristics. The corporation’s immense machine intelligence capabilities transform these data into vivid prediction products.