Why would these machines be so dangerous? To understand the threat, you need to shed some of your human biases about the scales of intellectual ability. As AI theorist Eliezer Yudkowsky puts it, we have a “human tendency to think of ‘village idiot’ and ‘Einstein’ as the extreme ends of the intelligence scale, instead of nearly indistinguishable points on the scale of minds-in-general.” From the point of view of, say, a mouse, the village idiot and Einstein are both unfathomably intelligent. We spent the first decades of AI research mostly dreaming about building machines that might function at a village-idiot level of intelligence or perhaps reach the Einsteinian summit. But as the philosopher Nick Bostrom and Yudowsky both argue, there’s no reason to think that the Einsteinian summit is some sort of fundamental upper limit. “Far from being the smartest possible biological species,” Bostrom writes, “we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.” Powered by recursive, self-learning algorithms, the first true AI might well march right past Mount Einstein and ascend to some higher plateau well beyond our imagining.
The danger perceived by people like Bostrom or Hawking does not look exactly like the standard science-fiction version. First, it is not at all necessary that the AI become conscious (or “self-aware,” as the original Terminator put it). A superintelligent AI might develop some kind of alternative consciousness, likely completely different from ours. But it also might remain a vast assemblage of insentient calculations, capable of expression and action and long-term planning, but lacking a sense of self. Secondly, the AI need not suddenly turn evil or vengeful or ambitious (or any other anthropomorphic emotion) to destroy human civilization. Bostrom, for instance, spends almost no time in his influential book Superintelligence imagining machines becoming evil overlords; instead, he worries about small miscommunications in defining the AI’s goals or motivations that could lead to global or even cosmic transformations. Consider programming an AI with as seemingly an innocuous goal as you could imagine: Bentham’s “greatest happiness for the greatest number.” You set that as the overarching value and let the machine decide the best approach to making it a reality. Maximizing human happiness would seem to be a perfectly laudable objective, but the AI might well come up with a scenario that, while technically achieving the objective, would be immediately abhorrent to humans: perhaps the AI distributes nanobots into every human brain on the planet, permanently stimulating the pleasure centers of the brain and turning us all into grinning zombies. The threat is not that when asked to decide the best strategy for combating some environmental crisis, the AI will actively disobey us and instead hack into the Department of Defense network and detonate its entire nuclear arsenal because it has evolved some inherent evilness or desire for conquest. The threat is that we will ask it to find the optimal solution for an environmental crisis, and it will decide to eliminate the main cause of the crisis—human beings—because we haven’t framed the objective clearly enough.
Much of the debate over superintelligent AI is devoted to thinking through what is sometimes called the “containment problem,” brilliantly explored in Alex Garland’s film Ex Machina: how to keep the genie of AI inside the bottle, while still tapping into its powers. Could humans evolve an AI that was truly superintelligent, but at the same time keep it safely bounded so that a runaway instruction doesn’t trigger a global catastrophe? In Bostrom’s convincing presentation, the problem is much harder than it might first appear, in large part because the humans would be trying to outthink an intelligence that is orders of magnitude more advanced than their own. Containing the AI will be like a mouse scheming to influence human technological advancement to prevent the invention of mousetraps.
In a way, we are at a point in the conversation about superintelligence equivalent to where the global warming debate was in the late 1980s: a small group of scientists and researchers and public intellectuals extrapolating out from current trends and predicting a major crisis looming several generations down the line. According to a survey conducted by Bostrom, most of the AI research community believes superhuman-level AI is still at least fifty years away.
That multigenerational time scale may be the most encouraging element in a debate filled with doomsday scenarios. Climate advocates often complain about the sluggish pace of political and corporate reform given the magnitude of the global warming threat. But we should remind ourselves that with climate change, we are trying to make a series of decisions that are arguably without precedent in human history: deciding which regulatory and technological interventions to put in place to prevent a threat that may not have a severe impact on most humans for several decades, if not longer. For all the biases and intuitive leaps of System 1, one of the hallmarks of human intelligence is the long-term decision-making of System 2: our ability to make short-term sacrifices in the service of more distant goals, the planning and forward thinking of Homo prospectus. While we are not flawless at it by any means, we are better at that kind of thinking than any other species on the planet. But we have never used those decision-making skills to wrestle with a problem that doesn’t exist yet, a problem we anticipate arising in the distant future based on our examination of current trends.
To be clear, humans have made decisions to engineer many ingenious projects with the explicit aim of ensuring that they last for centuries: pyramids, dynasties, monuments, democracies. Some infrastructure decisions—like the dike system of the Netherlands or Japanese building codes designed to protect against tsunamis—anticipate threats that might not happen for a century or more, though those threats are not genuinely new ones: those cultures know to be concerned about floods and tsunamis because they have experienced them in the past. Some decisions that we have made, like the decision to adopt democratic governance, have been explicitly designed to solve as-of-yet-undiscovered problems by engineering resilience and flexibility into their codes and conventions. But mostly those exercises in long-term planning have been all about preserving the current order, not making a preemptive choice to protect us against threats that might erupt three generations later. In a way, the closest analogues to the current interventions on climate (and the growing AI discussion) are eschatological: in religious traditions that encourage us to make present-day decisions based on an anticipated Judgment Day that may not arrive for decades, or millennia.
With superintelligence, as with climate change, we are trying something new as a species. We are actively thinking about the choices we are making now in order to achieve a better outcome fifty years from now. But superintelligence is an even more ambitious undertaking, because the problem we are anticipating is qualitatively different from today’s reality. Climate change forces us to imagine a world that is a few degrees hotter than our current situation, with longer droughts, more intense storms, and so on. We talk about global warming “destroying the planet,” but that language is hyperbole: the planet will be fine even if we do nothing to combat global warming. Even in the worst-case scenario, Homo sapiens as a species would survive a five-degree increase in surface temperatures—though not without immense suffering and mortality. A truly superintelligent machine—capable, for example, of building self-replicating nano-machines that devour all carbon-based life—could plausibly pose an extinction-level threat to us. But there is nothing in our current landscape or our history that resembles this kind of threat. We have to imagine it.
Interestingly, one of the key tools we have had in training our minds to make this momentous choice has been storytelling—science fiction, to be precise, which turns out to play a role in some of our mass decisions equivalent to the role scenario planning plays in our group decisions. “This kind of exercise is generally new,” the writer and futurist Kevin Kelly suggests, “because we all now accept that the world of our grandchildren will be markedly different than our world—whic
h was not true before. I believe this is the function of science fiction. To parse, debate, rehearse, question, and prepare us for the future of new. For at least a century, science fiction has served to anticipate the future . . . In the past there have been many laws prohibiting new inventions as they appeared. But I am unaware of any that prohibited inventions before they appeared. I read this as a cultural shift from science fiction as entertainment to science fiction as infrastructure—a necessary method of anticipation.” Science-fiction narratives have been ruminating on the pitfalls of artificial intelligence for at least a century—from the “global brain” of H. G. Wells to HAL 9000 all the way to Ex Machina—but only in the last few years has the problem entered into real-world conversation and debate. The novels primed us to see the problem more clearly, helped us peer around the limits of our technology-bounded rationality. No doubt the superintelligent machines will climb their way past human intelligence by running ensemble simulations at unimaginable speeds, but if we manage to keep them from destroying life as we know it, it will be, in part, because the much slower simulations of novels helped us understand the threat more clearly.
Given the accelerating rates of change of modern society, the current debate about AI and its potential threats is a bit like a group of inventors and scientists gathering together in the early 1800s and saying, “This industrial revolution is certainly going to make us much more productive, and in the long run raise standards of living, but we also appear to be pumping a lot of carbon into the atmosphere, which will likely come back to haunt us in a couple of centuries, so we should think about how to prevent that problem.” But, of course, that conversation didn’t happen, because we didn’t have the tools to measure the carbon in the air, or computer simulations to help us predict how that carbon would influence temperatures globally, or a history of battling other industrial pollutants, or government and academic institutions that monitor climate and ecosystem change, or sci-fi novels that imagined a scenario where new technologies somehow altered global weather patterns. We were smart enough to invent coal-powered engines, but not yet smart enough to predict their ultimate impact on the environment.
The AI debate is yet another reminder of how much progress we have made in our ability to make farsighted decisions. All those tools and sensors and narratives that have enabled us to identify the threat of climate change or imagine an AI apocalypse constitute, en masse, their own kind of superintelligence.
“We are as gods,” Stewart Brand famously wrote a half century ago. “We might as well get good at it.” We have indeed developed godlike powers over our planet’s atmosphere in just under three hundred years of carbon-based industry. Are we good at it yet, though? Probably not. But we’re quick learners. And we’re certainly taking on global decisions with time horizons that our immediate ancestors would have found astonishing. The fact that challenges are still emerging to long-term global decisions like the Paris Agreement is inevitable: it’s hard enough to project forward fifty years as an individual, much less as a society. But just the existence of these debates—AI, climate change, METI—make it clear that we are beginning to explore a new kind of farsightedness. With AI, all the projections of future threats may well turn out to be false alarms, either because true AI turns out to be far more difficult to achieve, or because we discover new techniques that minimize the danger before the machines march on past Mount Einstein. But if artificial superintelligence does turn out to pose an existential threat, our best defense will likely come out of our own new powers of human superintelligence: mapping, predicting, simulating, taking the long view.
THE DRAKE EQUATION
Superintelligence, climate change, and METI share another property beyond their extended time horizons. They are all decisions that cannot be properly appraised without the consultation of a wide range of intellectual disciplines. Climate science alone is a hybrid of multiple fields: molecular chemistry, atmospheric science, fluid dynamics, thermodynamics, hydrology, computer science, ecology, and many more. Defining the problem of climate change didn’t just require the digital simulations of Cheyenne; it also required a truly heroic collaboration between disciplines. But deciding what to do about climate change requires a whole other set of fields as well: political science, economics, industrial history, and behavioral psychology, for instance. The problem of superintelligence draws on expertise in artificial intelligence, evolution, and software design, but it also has been profoundly illuminated by philosophical inquiries and the imagined futures of science fiction. Some amount of intellectual diversity is required in any full-spectrum decision, of course; even the most intimate choice, as we will see in the next chapter, draws on multiple bands of experience to settle on an optimal path. But these mass decisions—the ones that may well involve existential risk to us as a species—require an even wider slice of the spectrum.
A little more than a decade before he transmitted his famous Arecibo Message—the one that cannot, by definition, receive a reply for another hundred thousand years—Frank Drake sketched out one of the great equations in modern scientific history, as a way of framing the decision of whether to seek contact with lifeforms on other planets. If we start scanning the cosmos for signs of intelligent life, Drake asked, how likely are we to actually detect something? The equation didn’t generate a clear answer; it was more of an attempt to build a full-spectrum map of all the relevant variables. In mathematical form, the Drake equation looks like this:
N = R* × ƒp × ne × ƒl × ƒi × ƒc × L
N represents the number of extant, communicative civilizations in the Milky Way. The initial variable R* corresponds to the rate of star formation in the galaxy, effectively giving you the total number of potential suns that could support life. The remaining variables then serve as a kind of nested sequence of filters: Given the number of stars in the Milky Way, what fraction of those have planets, and how many of those have an environment that can support life? On those potentially hospitable planets, how often does life itself actually emerge, and what fraction of that life evolves into intelligent life, and what fraction of that life eventually leads to a civilization’s transmitting detectable signals into space? At the end of his equation, Drake placed the crucial variable L, which is the average length of time during which those civilizations emit those signals.
I know of no other equation that so elegantly conjoins so many different intellectual disciplines in a single framework. As you move from left to right in the equation, you shift from astrophysics, to the biochemistry of life, to evolutionary theory, to cognitive science, all the way to theories of technological development. Your guess about each value in the Drake Equation winds up revealing a whole worldview. Perhaps you think life is rare, but when it does emerge, intelligent life usually follows; or perhaps you think microbial life is ubiquitous throughout the cosmos, but more complex organisms almost never form. The equation is notoriously vulnerable to very different outcomes, depending on the numbers you assign to each variable.
The most provocative value is the last one: L, the average life span of a signal-transmitting civilization. You don’t have to be a Pollyanna to defend a relatively high L value. You just have to believe it’s possible for civilizations to become fundamentally self-sustaining and survive for millions of years. Even if one in a thousand intelligent life-forms in space generates a million-year civilization, the value of L increases meaningfully. But if your L value is low, that implies a further question: What is keeping it low? Do technological civilizations keep flickering on and off in the Milky Way, like so many fireflies in space? Do they run out of resources? Do they blow themselves up?
Since Drake first sketched out the equation in 1961, two fundamental developments have reshaped our understanding of the problem. First, the product of the first three values in the equation (representing our best guess at the number of stars with habitable planets) has increased by several orders of magnitude. And second, we have been listening for signals
for decades and heard nothing. If the habitable planet value keeps getting bigger and bigger without any sign of intelligent life in our scans, the question becomes: Which of the other variables are the filters? Perhaps life itself is astonishingly rare, even on habitable planets. From our perspective, as human beings living in the first decades of the third millennium, wondering whether we are flirting with existential risks through our technological hubris, we want the emergence of intelligent life to be astonishingly rare; if the opposite is true, and intelligent life is abundant in the Milky Way, then L values might be low, perhaps measured in centuries and not even millennia. In that case, the adoption of a technologically advanced lifestyle might be effectively simultaneous with extinction. First you invent radio, then you invent technologies capable of destroying all life on your planet and shortly thereafter you push the button and your civilization goes dark.
Perhaps this is the ironic fate of any species that achieves the farsightedness of Homo prospectus. Perhaps every time a species on some Earth-like planet evolves a form of intelligence smart enough to imagine alternate futures, smart enough to turn those imaginative acts into reality, that cognitive leap forward sets off a chain reaction of technological escalation that ultimately deprives that species of its actual future. The early silence that has greeted our SETI probes so far suggests that this is at the very least a possibility. But perhaps that escalation is an arms race that is not doomed to end in apocalypse. Maybe the L values are high, and the universe is teeming with intelligent life that made it through the eye of the needle of industrialization without catastrophe. Maybe it’s possible to invent ways of making farsighted choices as a society faster than we invent new ways of destroying ourselves. Certainly it’s essential for us to try. If those superintelligent machines do manage to assist human civilization, and not accidentally trigger the mass extinction that Bostrom and Hawking fear, it will be because those machines learned how to make decisions that assessed the full spectrum of variables and consequences, that ran ensemble simulations that allowed them to tease out all the unanticipated consequences and discover new options. Perhaps the machines will evolve that farsightedness on their own, through some kind of self-learning algorithm. But wouldn’t it be better if we were wise enough by then to give them a head start?
Farsighted Page 16