The Age of Surveillance Capitalism

by Shoshana Zuboff

  That Pentland’s “we” is able “to generate social pressure for change” reflects his understanding of the superorganism. The God view persuades him that assessing human action really is comparable to counting beavers: “We can observe humans in just the same way we observe apes or bees and derive rules of behavior, reaction, and learning.”63 In all of these populations, the collective exerts pressure on each organism to go with the flow, stay with the herd, return to the hive, and take flight with the flock. Idea flows mimic the pattern of the machine hive, the edge conflates with the hub, identity yields to and synchrony, the parts dissolve in the whole. He writes:

  I believe that we can we think of each stream of ideas as a swarm or collective intelligence, flowing through time, with all the humans in it learning from each other’s experiences in order to jointly discover the patterns of preferences and habits of action that best suit the surrounding physical and social environment. This is counter to the way most modern Westerners understand themselves, which is as rational individuals, people who know what they want and who decide for themselves what actions to take in order to accomplish their goals.64

  This shift from society to swarm and from individuals to organisms is the cornerstone upon which the structure of an instrumentarian society rests.

  Pentland ignores the role of empathy in emulation because empathy is a felt experience that is not subject to the observable metrics required for computational governance. Instead, Pentland subscribes to the label Homo imitans to convey that it is mimicry, not empathy, and certainly not politics, which defines human existence. The term itself derives from studies of infant learning, but for Pentland it is a fitting explanation of all human behavior all the time: an assertion, like Skinner’s, that control always rests with society. “The largest single factor driving adoption of new behaviors,” he writes, is “the behavior of peers.”65

  Because we are born to imitate one another, Pentland argues, the whole species is attuned to social pressure as an efficient means of behavioral modification. This model of human learning is a throwback to bees and apes but also a forward pass to the machine hive. Machines do not learn by empathy; learning is automatically updated in a lockstep progression of collective intelligence.

  4. Applied Utopistics

  Both Skinner and Pentland believe in the authority of the utopianists to impose their plan. Instrumentarian society is a planned society, produced through total control of the means of behavioral modification. Neither Skinner’s planners nor Pentland’s tuners shrink from their responsibility to wield the power that shapes the superorganism.

  Skinner never lost faith in the social vision of Walden Two. He understood utopia as a “total social environment” in which all parts work in harmony toward collective aims:

  The home does not conflict with the school or the street, religion does not conflict with government.… And if planned economies, benevolent dictatorships, perfectionistic societies, and other utopian ventures have failed, we must remember that unplanned, undictated, and unperfected cultures have failed too. A failure is not always a mistake; it may simply be the best one can do under the circumstances. The real mistake is to stop trying.66

  Pentland similarly conceives his social physics as both comprehensive and necessary. Nothing short of its totalistic rendering and control of all human behavior will serve civilization in a hyperconnected future, and there is no sign of hesitation to assert computational governance over the whole domain of human endeavor for the sake of a collective destiny. The politics and economics of that destiny, which is to say the authority and power that found and sustain it, do not require specification because the machines and their math transcend these once fundamental coordinates of human society. Instead, computation reveals the truth hidden in the data and therefore determines what is “correct.” A new social class of tuners exercises perpetual vigilance to cure human nature of its weaknesses by ensuring that populations are tuned, herded, and conditioned to produce the most-efficient behaviors. The “tools of social network incentives” are all that is required “to establish new norms of behavior, rather than relying on regulatory penalties and market competition.… Given the well-known shortcomings of human nature, social efficiency is a desirable goal.… Our focus should be on providing the idea flow required for individuals to make correct decisions and develop useful behavioral norms.…”67 Finally, like Skinner, Pentland rejects the notion that his imagined “data-driven society” is merely a utopian fantasy, insisting instead that it is not only practical and feasible but also a moral imperative in which the benefits to the collective outweigh all other considerations.

  5. The Death of Individuality

  Individuality is a threat to instrumentarian society, troublesome friction that sucks energy from “collaboration,” “harmony,” and “integration.” In an article titled “The Death of Individuality,” Pentland insists that “instead of individual rationality, our society appears to be governed by a collective intelligence that comes from the surrounding flow of ideas and examples.… It is time that we dropped the fiction of individuals as the unit of rationality and recognised that our rationality is largely determined by the surrounding social fabric.…”68

  Here again the behaviorist from Harvard had already sounded the first and most eloquent iteration of this message, elevating the Other-One and denouncing the autonomous self. In Beyond Freedom & Dignity, Skinner freely displayed his contempt for this most transcendent ideal of the Sartrean age: the will to will oneself into first-person voice and action. Skinner argued that the differences between humans and other species are greatly exaggerated, and he would have found Pentland entirely justified in his rejection of the individual in favor of the distant, computer-mediated gaze. Beavers or people, the variance hardly matters once we shed the destructive fiction of individual autonomy. The surrender of the individual to manipulation by the planners clears the way for a safe and prosperous future built on the forfeit of freedom for knowledge. Skinner was unrelenting on this point:

  What is being abolished is autonomous man—the inner man, the homunculus, the possessing demon, the man defended by the literatures of freedom and dignity. His abolition has long been overdue.… He has been constructed from our ignorance, and as our understanding increases, the very stuff of which he is composed vanishes… and it must do so if it is to prevent the abolition of the human species. To man qua man we readily say good riddance. Only by dispossessing him can we turn… from the inferred to the observed, from the miraculous to the natural, from the inaccessible to the manipulable.69

  The long-overdue death of individuality finally dispels the distracting fictions that fetishize the notions of freedom and dignity. The twentieth-century behaviorist from Harvard and the twenty-first-century data scientist from MIT agree that the notion of free will is but another unfortunate hangover from a dark age when science had not yet demonstrated that, as Skinner says, we live “under the control of a social environment” that “millions of others… have constructed.” The blunt behaviorist delivers his final truth: “A person does not act upon the world, the world acts upon him.”70

  In a lecture at Google that garnered enthusiastic applause, Pentland flattered the audience by signaling that the digital cognoscenti will easily accept the obsolescence of the individual as a necessary fate. “What about free will?” he asked the audience in Mountain View. “That may not have occurred to you, but that’s a traditional thing to ask.” He went on to explain that most human behavior—from political views to spending choices to the music that people listen to—is predicted by “what’s cool to do… exposure to what other people do.” Many people reject this idea, he noted, because “it’s not the rhetoric in our society.” Then he assured the Googlers, “You guys are the last people to be saying this to, because you guys are like the best and smartest in the world.” For such people as these, Pentland appeared to say, the death of the individual is yesterday’s news:

  So you’ve heard about r
ational individuals. And everybody rags on the rational part. I’m not going to do that. I’m going to rag on the individual part, OK? Because I don’t think we are individuals. What we desire, the ways we learn to go about doing it, what’s valuable, are consensual things… individual incentives… that’s part of this mindset that comes from the 1700s… the action is not between our ears. The action is in our social networks, OK? We are a social species.71

  Pentland’s vision is Skinner’s vision, now standing on the shoulders of Big Other with its Big Data and its Big Math. These are the resources of the smart machines required to divine the “correct” answers. Such is Pentland’s resonance with Skinner’s social theory that without ever mentioning the behaviorist’s name, a later section of Pentland’s book is titled “Social Physics Versus Free Will and Dignity.”

  If we are to annihilate and bury the individual as an existential reality, philosophical idea, and political ideal, then this death should at least merit the gravitas of an ancient Greek funerary ritual. The existence of the individual is, after all, an achievement carved from millennia of human suffering and sacrifice. Instead, Pentland brushes it aside as just another debugging of humanity’s computer code, a much-needed upgrade to the outdated software that is the long human story.

  Unlike Skinner, though, Pentland is careful to round the square, perhaps in the hope of evading a withering review by the likes of Noam Chomsky. (In “The Case Against B. F. Skinner,” as you may recall from Chapter 10, Chomsky famously denounced Skinner as “vacuous” and “devoid of scientific content,” and he assessed the work as burdened with misconceptions that “virtually guarantee failure.”72) Pentland sidesteps the risks of Skinner’s jeremiad by taking a softer tone: “Some people react negatively to the phrase social physics, because they feel that it implies that people are machines without free will and without the ability to move independently of our role in society.”73 Like Meyer, Pentland acknowledges that humans have a “capacity for independent thought” but insists that social physics “does not need to try to account for it.” As Pentland sees it, the problem is not that “independent thought” is omitted from the picture but rather that “internal, unobservable” thought processes are just friction that “will occasionally emerge to defeat our best social physics models.” Fortunately, the models are not really in danger because “the data tell us that deviations from our regular social patterns occur only a few percent of the time.”74 The autonomous individual is but a statistical blip, a slip of the pen that is easily overridden in the march toward confluent action and someone’s greater good.

  In this vein, Professor Pentland does not ignore issues like privacy and social trust. He actively advocates solutions to these problems, but the solutions he pursues are already tailored to the intensification of a “data-driven” instrumentarian society. Pentland’s approach recalls the early conviction of his former doctoral student Rosalind Picard that societal challenges are not insurmountable, that new technical solutions will resolve any problems, and that “safeguards can be developed.” Two decades later Picard’s view had a darker cast, but Pentland expresses little trace of doubt. For example, Pentland works with influential institutions like the World Economic Forum to craft “a new deal on data” that favors individual “ownership” of personal information but does not question the ubiquitous rendition of such personal information in the first instance.75 Data ownership, he believes, will create financial incentives for participation in a market-oriented instrumentarian society. Like Skinner, Pentland assumes that the sheer weight of incentives and ubiquitous connection, monitoring, and tuning will eventually wear down older sensibilities such as the interest in privacy. “The New Deal gives customers a stake in the new data economy; that will bring first greater stability and then eventually greater profitability as people become more comfortable sharing data.”76

  In Pentland’s view of data ownership, certainty machines like blockchain, which relies on complex encryption and algorithms to create a decentralized tamper-proof database, are commandeered to bypass social trust. He advocates systems “that live everywhere and nowhere, protecting and processing the data of millions of people, and executing on millions of internet computers.”77 One important study of Bitcoin, the cryptocurrency that relies on blockchain, suggests that such machine solutions both express and contribute to the general erosion of the social fabric in ways that are both consistent with instrumentarianism and further pave the way for its success. Information scholars Primavera De Filippi and Benjamin Loveluck conclude that contrary to popular belief, “Bitcoin is neither anonymous nor privacy-friendly… anyone with a copy of the blockchain can see the history of all Bitcoin transactions… every transaction ever done on the Bitcoin network can be traced back to its origin.” Such systems rely on “perfect information,” but the kinds of coordination processes that build open democratic societies, such as “social trust” or “loyalty,” are “expunged” in favor of “a profoundly market-driven approach.”78 Like Varian, Pentland does not acknowledge the social and political implications of such systems, which are in any case irrelevant to an instrumentarian future in which democracy and social trust are superseded by the certainty machines, their priests, and their owners.

  Surveillance capitalism grew to dominance during the years that Pentland has argued for his “New Deal,” even as it benefitted from his theoretical and commercial innovations. During those same years, as we have seen, Picard’s “affective computing” fell to the surveillance paradigm. Nevertheless, Professor Pentland is sanguine that surveillance capitalism can easily be pushed aside by market forces, despite its concentrations of knowledge, rights, and power; its unilateral control of the shadow text; and its dominant position in the division of learning in society. “It simply requires that creative businesspeople harness the will of consumers in order to construct a value proposition better than the current steal-all-your-data paradigm. We’ve just got to push on through.”79 Power, politics, and law do not enter into the equation, presumably because they are already obsolete in the social vision under construction here.

  IV. The Third Modernity of the Hive

  It is no surprise that capitalism shapes social relations. A century ago it was the new means of mass production that fashioned mass society in its image. Today, surveillance capitalism offers a new template for our future: the machine hive in which our freedom is forfeit to perfect knowledge administered for others’ profit. This is an unheralded social revolution that is difficult to discern in the fog of utopian rhetoric and high-speed applied utopistics conjured by leading surveillance capitalists and the many communities of practice—from developers to data scientists—that enable and sustain the dominance of the commercial surveillance project.

  Surveillance capitalists work hard to camouflage their purpose as they master the uses of instrumentarian power to shape our behavior while evading our awareness. That is why Google conceals the operations that turn us into the objects of its search and Facebook distracts us from the fact that our beloved connections are essential to the profit and power that flow from its network ubiquity and totalistic knowledge.

  Pentland’s experimental work and theoretical analyses perform an important political and social function in piercing this fog. They map the tactical and conceptual pathways of instrumentarian society that place the means of behavior modification at the heart of this social system, founded on the scientific and technological control of collective behavior and administered by a specialist class. In China the state appears determined to “own” this complex, but in the West it is largely owned and operated by surveillance capital.

  Instrumentarian society defines the ultimate institutionalization of a pathological division of learning. Who knows? Who decides? Who decides who decides? Here too the comparison with China is useful. An abnormal division of learning marks both China and the West. In China the state vies with its surveillance capitalists for control. In the US and Europe the state works with and through
the surveillance capitalists to accomplish its aims. It is the private companies who have scaled the rock face to command the heights. They sit at the pinnacle of the division of learning, having amassed unprecedented and exclusive wealth, information, and expertise on the strength of their dispossession of our behavior. They are making their dreams come true. Not even Skinner could have aspired to this condition.

  The social principles of instrumentarianism’s third modernity represent a stark break with the legacies and ideals of the liberal order. Instrumentarian society is a topsy-turvy fun-house-mirror world in which everything that we have cherished is turned upside down and inside out. Pentland doubles down on the illiberality of behavioral economics. In his hands the ideology of human frailty is not merely cause for contempt but a justification for the death of individuality. Self-determination and autonomous moral judgment, generally regarded as the bulwark of civilization, are recast as a threat to collective well-being. Social pressure, well-known to psychologists for its dangerous production of obedience and conformity, is elevated to the highest good as the means to extinguish the unpredictable influences of autonomous thought and moral judgment.