Oxford University China scholar Rogier Creemers, who translated some of the first documents on the social credit system, observes that “the trend towards social engineering and ‘nudging’ individuals towards ‘better’ behavior is also part of the Silicon Valley approach that holds that human problems can be solved once and for all through the disruptive power of technology.… In that sense, perhaps the most shocking element of the story is not the Chinese government’s agenda, but how similar it is to the path technology is taking elsewhere.”46
In 2017 a surveillance technology trade show held in Shenzhen was packed with US companies selling their latest wares, especially cameras equipped with artificial intelligence and facial recognition. Among the crowd was the managing director of CCTV direct, a UK distributor of surveillance equipment. He lamented “how far behind the Western countries are,” compared to the skills and thrills of China’s surveillance infrastructure, but he also comforted himself with this thought: “What starts here ends up in homes, airports, and businesses back in America.”47
The difference between surveillance capitalism in the West and China’s emerging social credit system pivots on the patterns of entanglement and engagement between instrumentarian and state power. There are structural differences. In the West, as we have seen, the patterns have taken on many forms. The state began as bosom and shelter, then eager student and envious cousin. Surveillance capitalism and its instruments have come of age now, producing a fitful but necessary partnership. Key instrumentarian capabilities are docked in the big surveillance capitalist firms, and the state must move with and through these firms to access much of the power it seeks.
In the Chinese context, the state will run the show and own it, not as a market project but as a political one, a machine solution that shapes a new society of automated behavior for guaranteed political and social outcomes: certainty without terror. All the pipes from all the supply chains will carry behavioral surplus to this new, complex means of behavioral modification. The state will assume the role of the behaviorist god, owning the shadow text and determining the schedule of reinforcements and the behavioral routines that it will shape. Freedom will be forfeit to knowledge, but it will be the state’s knowledge that it exercises, not for the sake of revenue but for the sake of its own perpetuation.
V. A Fork in the Road
Recall Carl Friedrich’s observation on the challenge of grasping the naked facts of totalitarianism: “Virtually no one before 1914 anticipated the course of development which has overtaken Western civilization since then.… To this failure to foresee corresponds a difficulty in comprehending.”48 Recall too the grinning, robust “Joe” Stalin planted among Hollywood luminaries on the glossy pages of a 1939 Look magazine. Will we suffer the same lack of foresight as those who could not comprehend totalitarianism’s rise, paralyzed by the sheer power of Big Other and its infinite echoes of consequence, distracted by our needs and confused by its speed, secrecy, and success?
Astonishment is a necessary alarm. We need it, but it should not leave us frozen in disbelief. The steady drumbeat of Big Other’s manifest destiny, its breathtaking velocities, and the obscurity of its aims and purpose are intended to disarm, disorient, and bewilder. Inevitabilist ideology works to equate surveillance capitalism and its instrumentarian power with nature: not a human construction but something more like a river or a glacier, a thing that can only be joined or endured. All the more reason to ask: Might the banalities of today’s declarations (“instruct the vehicular monitoring system not to allow the car to be started”) also reveal themselves in the fullness of time as the seeds of our century’s greatest nightmare? What of the authors of the instrumentarian project? How will we appraise the smiling, robust faces of the tech titans when we revisit those images in the glossy pixels of some twenty-first-century version of Look? The road from Shenzhen to an American or European airport also leads to the Roomba vacuum cleaner mapping your living room and your breakfast with Alexa. It is the road to machine certainty imposed by instrumentarian power and produced by surveillance capitalism. This journey is not as long as you might think.
There is a fork in the road.
In one direction lies the possibility of a synthetic declaration for a third modernity based on the strengthening of democratic institutions and the creative construction of a double movement for our time. On this road we harness the digital to forms of information capitalism that reunite supply and demand in ways that are both genuinely productive of effective life and compatible with a flourishing democratic social order. The first step down this road begins with naming, establishing our bearings, reawakening our astonishment, and sharing a sense of righteous indignity.
If we follow the other road, the one that links us to Shenzhen, we find our way to surveillance capitalism’s antidemocratic vision for a third modernity fashioned by instrumentarian power. It is a future of certainty accomplished without violence. The price we pay is not with our bodies but with our freedom. This future does not yet exist, but like Scrooge’s dream of Christmas future, the materials are all in place and ready for assembly. Chapter 14 examines this next way station on the road that began with an unprecedented capitalism, turned toward an unprecedented power, and now leads to an unprecedented society, theorized and legitimated by a burgeoning intellectual ecosystem of thinkers, researchers, and practitioners. What is this new place that they would have us call home?
Figure 4: Two Species of Power
CHAPTER FOURTEEN
A UTOPIA OF CERTAINTY
So from the years their gifts were showered: each
Grabbed at the one it needed to survive;
Bee took the politics that suit a hive,
Trout finned as trout, peach molded into peach,
And were successful at their first endeavor.
—W. H. AUDEN
SONNETS FROM CHINA, I
I. Society as the Other-One
Although he did not name it, the visionary of ubiquitous computing, Mark Weiser, foresaw the immensity of instrumentarian power as a totalizing societal project. He did so in a way that suggests both its utter lack of precedent and the danger of confounding it with what has gone before: “hundreds of computers in every room, all capable of sensing people near them and linked by high-speed networks have the potential to make totalitarianism up to now seem like sheerest anarchy.”1 In fact, all those computers are not the means to a digital hyper-totalitarianism. They are, as I think Weiser sensed, the foundation of an unprecedented power that can reshape society in unprecedented ways. If instrumentarian power can make totalitarianism look like anarchy, then what might it have in store for us?
Seven decades ago, Skinner’s proto-instrumentarian behavioral utopia, Walden Two, was met with revulsion. Today the real thing is inspirational fodder for surveillance capitalist rhetoric as leaders promote the tools and visions that will bring the old professor’s ideas to life… to our lives. The processes of normalization and habituation have begun. We have already seen that surveillance capitalism’s pursuit of certainty—the mandate of the prediction imperative—requires a continuous approximation to total information as the ideal condition for machine intelligence. On the trail of totality, surveillance capitalists enlarged their scope from the virtual to the real world. The reality business renders all people, things, and processes as computational objects in an endless queue of equivalence without equality. Now, as the reality business intensifies, the pursuit of totality necessarily leads to the annexation of “society,” “social relations,” and key societal processes as a fresh terrain for rendition, calculation, modification, and prediction.
Big Other’s ubiquity is revered as inevitable, but that is not the endgame. The aim in this new phase is the comprehensive visibility, coordination, confluence, control, and harmonization of social processes in the pursuit of scale, scope, and action. Although instrumentarianism and totalitarianism are distinct species, they each yearn toward totality, though in profoundly different
ways. Totalitarianism seeks totality as a political condition and relies on violence to clear its path. Instrumentarianism seeks totality as a condition of market dominance, and it relies on its control over the division of learning in society, enabled and enforced by Big Other, to clear its path. The result is the application of instrumentarian power to societal optimization for the sake of market objectives: a utopia of certainty.
Although they resonate in many respects with the instrumentarian social vision of China’s political elite, surveillance capitalists have distinct objectives. In their view, instrumentarian society is a market opportunity. Any norms and values they impose are designed to further the certain fulfillment of market goals. Like human experience, society is subordinated to the market dynamic and reborn as objectified computational behavioral metrics available to surveillance capitalism’s economies of scale, scope, and action in the pursuit of the most-lucrative supplies of behavioral surplus. In order to achieve these aims, surveillance capitalists have conjured a chilling vision. They aim to fashion a new society that emulates machine learning in much the same way that industrial society was patterned on the disciplines and methods of factory production. In their vision, instrumentarian power replaces social trust, Big Other substitutes certainty for social relations, and society as we know it shades into obsolescence.
II. Totality Includes Society
Like generals delivering a chest-thumping tally of their armies, surveillance capitalist leaders take care to assure allies of their great power. This is typically expressed in an inventory of the instrumentarian troops massed at the border, poised for the rendition of everything in pursuit of totality. This pursuit, it becomes clear, does not merely have consequences for society; it includes society.
In the spring of 2017, Microsoft CEO Satya Nadella bounded onstage to open the company’s annual developers’ conference, his slender profile accentuated by the requisite black polo shirt, black jeans, and trendy black high-tops. He quickly dazzled the audience with a roll call of his troops. Nadella recounted the 500 million Windows 10 devices; 100 million monthly users of its Office software; 140 million monthly users of the corporation’s digital “assistant,” Cortana; and more than 12 million organizations signed on to its cloud services, including 90 percent of the Fortune 500.
Nadella did not fail to remind his audience of the crushing velocity that drives the instrumentarian project in an explosion of shock and awe, especially in the years since surveillance capitalism came to dominate digital services: internet traffic increased by a factor of 17.5 million over 1992’s 100 gigabytes per day; 90 percent of the data in 2017 was generated in the prior two years; a single autonomous car will generate 100 gigabytes per second; there will be an estimated 25 billion intelligent devices by 2020. “It’s stunning to see the progress across the depth and breadth of our society and economy and how digital technology is so pervasive.… It’s about what you can do with that technology to have broad impact.” His final exhortation to the assembled developers—“Change the world!”—earned a thunderous round of applause.2
In celebrating Google’s ambitions with the company’s developers in 2017, CEO Sundar Pichai ran parallel to Nadella, showcasing his troop strength as Google’s battalions fan out to embrace every corner of social life, demonstrating the breadth and depth of the corporation’s instrumentarian power with a zeal that would have made Professor Skinner glow. Pichai reports that seven of the company’s most-salient “products and platforms” engage one billion monthly active users, including Gmail, Android, Chrome, Maps, Search, YouTube, and the Google Play Store; two billion active Android devices; 800 million monthly active users of Google Drive with three billion objects uploaded each week; 500 million Photos users uploading 1.2 billion photos each day; 100 million devices using Google Assistant. Every device is recast as a vehicle for Assistant, which will be available “throughout the day, at home and on the go” for every kind of task or social function. Pichai wants even more, telling his team, “We must go deeper.” Assistant should be wherever “people might want to ask for help.” Google executives share the enthusiasm. “Technology is now on the cusp of taking us into a magical age,” writes Eric Schmidt, “solving problems today that we simply couldn’t solve on our own.”3 Machine learning, he says, will do everything from curing blindness to saving animals from extinction. Above all, however, it is founder Larry Page who has long had his sights set on the transformation of society.
“The societal goal is our primary goal,” Page told the Financial Times in 2016.4 “We need revolutionary change, not incremental change,” he told another interviewer that year. “We could probably solve a lot of the issues we have as humans.”5 Much of Page’s future vision turns out to be stock utopian fare, themes that have been repeated for millennia. Page anticipates machine intelligence that restores humankind to the Garden of Eden, lifting us from toil and struggle into a new realm of leisure and fulfillment. He foresees, for example, a future society graced by “abundance” in all things, where employment is but a “crazy” distant memory.6
Most unusual, however, is that Page portrays Google’s totalistic ambitions as a logical consequence of its commitment to the perfection of society. From his point of view, we should welcome the opportunity to lean on Big Other and willingly subordinate all knowledge and decision rights to Google’s plan. For the sake of the plan, the totality of society—every person, object, and process—must be corralled into the supply chains that feed the machines, which, in turn, spin the algorithms that animate Big Other to manage and mitigate our frailty:
What you should want us to do is to really build amazing products and to really do that… we have to understand apps and we have to understand things you could buy, and we have to understand airline tickets. We have to understand anything you might search for. And people are a big thing you might search for.… We’re going to have people as a first class object in search… if we’re going to do a good job meeting your information needs, we actually need to understand things and we need to understand things pretty deeply.7
Total knowledge is sold as a requirement of the “preemptive” services that lead to the solution of solutions in the AI-powered, omniscient “Google Assistant”:
It’s really trying to understand everything in the world and make sense of it.… A lot of queries are actually about places, so we need to understand places.… A lot of the queries are about content we can’t find. We did books, and so on.… So, we’ve been gradually expanding that… maybe you don’t want to ask a question. Maybe you want to just have it answered for you before you ask it. That would be better.8
Google originated in the prospect of optimally organizing the world’s information, but Page wants the corporation to optimize the organization of society itself: “In my very long-term worldview,” he said in 2013, “our software understands deeply what you’re knowledgeable about, what you’re not, and how to organize the world so that the world can solve important problems.”9
Facebook CEO Mark Zuckerberg shares these totalistic ambitions, and he is increasingly frank about “society,” not just the individuals within it, as subordinate to Facebook’s embrace. His “three big company goals” include “connecting everyone; understanding the world; and building the knowledge economy, so that every user will have ‘more tools’ to share ‘different kinds of content.’”10 Zuckerberg’s keen appreciation of second-modernity instabilities—and the yearning for support and connection that is among its most-vivid features—drives his confidence, just as it did for Google economist Hal Varian. The corporation would know every book, film, and song a person had ever consumed. Predictive models would enable the corporation to “tell you what bar to go to” when you arrive in a strange city. The vision is detailed: when you arrive at the bar, the bartender has your favorite drink waiting, and you’re able to look around the room and identify people just like you.
Zuckerberg described the flow of behavioral surplus as “growing at an exponential ra
te… that lets us project into the future… two years from now people are going to be sharing twice as much… four years, eight times as much.…” And in a nod to the already pressing competition for totality, Zuckerberg anticipated that Facebook’s social graph will “start to be a better map of how you navigate the web than the traditional link structure.”11
To that end, the CEO told investors that Facebook would bring affordable internet access “to every person in the world” so that every user will have “more tools” to share “different kinds of content.”12 Nothing was likely to impede the corporation’s progress on the societal front, he asserted, because “humans have such a deep desire to express themselves.”13
In 2017 Zuckerberg went even further in articulating his societal ambitions, this time aiming straight at the heart of second-modernity anxieties: “People feel unsettled. A lot of what was settling in the past doesn’t exist anymore.” Zuckerberg believes that he and his company can provide a future “that works for everyone” and fulfills “personal, emotional, and spiritual needs” for “purpose and hope,” “moral validation,” and “comfort that we are not alone.” “Progress now requires humanity coming together not just as cities or nations,” Zuckerberg urged, “but also as a global community… the most important thing we at Facebook can do is develop the social infrastructure… to build a global community.…” Citing Abraham Lincoln, Facebook’s founder located his company’s mission in the evolutionary time line of civilization, during which humanity organized itself first in tribes, then cities, then nations. The next phase of social evolution would be “global community,” and Facebook was to lead the way, constructing the means and overseeing the ends.14
The Age of Surveillance Capitalism Page 49