by Paul Scharre
The science fiction theme of artificial humans rebelling against their makers is so common it has become known as the “Frankenstein complex,” after Mary Shelley’s nineteenth-century horror novel, Frankenstein. In it, Dr. Frankenstein, through the miracles of science, creates a humanlike creature cobbled together from leftover parts from “the dissecting room and the slaughter-house.” The monster turns on Dr. Frankenstein, stalking him and eventually murdering his new bride.
The fear that hubris can lead to uncontrollable creations has ancient roots that predate even Frankenstein. Jewish legend tells of a creature called a golem, molded from clay and brought to life by placing a shem, a Hebrew inscription containing one of the names of God, on the golem. In one such legend, Rabbi Judah Loew ben Bezalel of Prague molded a golem from the clay of Prague’s riverbanks in the sixteenth century to protect the Jewish community from anti-Semitic attacks. Golems, unlike later intelligent creations, were powerful but stupid beings that would slavishly follow orders, often to the detriment of their creators. Golem stories often end with the golem killing its creator, a warning against the hubris of playing God.
Human-level or superhuman AI tap into this deep well of fear of artificial beings. Micah Clark, a research scientist from the Florida Institute for Human & Machine Cognition who studies AI, cognition, and theory of mind, told me that at “a very personal and philosophical level, AI has been about building persons. . . . It’s not about playing chess or driving cars.” He explained, “With the general track of robotics and autonomous systems today, you would end up with autonomous systems that are capable but very, very dumb. They would lack any real sense of intelligence. They would be effectively teleoperated, just at a higher level of commanding.” Artificial general intelligence—what Clark calls “the dream of AI”—is about “personhood.”
Clark’s vision of AGI isn’t a fearful one, however. He envisions “the kind of persons that we would have intellectual, social, and emotional relationships with, that can experience life with us.” AI has been a lifelong passion for Micah Clark. As a child, he played computer games in his grandfather’s accounting office and a chess program in particular captured his imagination. The chess AI “destroyed” him, Clark said, and he wanted it to be able to teach him how to play better. Clark was looking for more than just a game, though. “I saw this potential for entertainment and friendship there, but the interaction side was pretty weak,” he explained. In college, Clark worked at NASA’s Jet Propulsion Laboratory on a large-scale AI demonstration project and he was hooked. Clark went on to study long-duration autonomy for interplanetary robotic spacecraft, but his research interests have moved beyond robotics, sensing, and actuation. The books on Clark’s desk in his office have titles like An Anatomy of the Mind and Consciousness and the Social Brain. Clark described the goal of AI research as “building human-like persons that can participate in human physical and social spaces and relationships.” (Clark is currently working for the Office of Naval Research and he is quick to caveat that these are not the goals of AI research in the Navy or the Department of Defense. Rather, these are the goals of the field of AI research as a whole.)
Clark’s vision of the future of AI is less Terminator and more like the movie Her. In Her, Joaquin Phoenix plays an awkward loner named Theodore who starts a relationship with an AI operating system called “Samantha.” Theodore and Samantha develop a close bond and fall in love. Theodore is shaken, however, when Samantha admits that she is simultaneously carrying on relationships with thousands of other people and is also in love with 641 of them. When Theodore breaks down, telling her “that’s insane,” she tries to lovingly explain, “I’m different from you.”
The otherness of artificial persons—beings like humans, but also fundamentally different—is a source of much of the fear of AI. Clark explained that AIs will need the ability to interact with humans and that involves abilities like understanding natural language, but that doesn’t mean that the AI’s behavior or the underlying processes for their intelligence will mirror humans’. “Why would we expect a silica-based intelligence to look or act like human intelligence?” he asked.
Clark cited the Turing test, a canonical test of artificial intelligence, as a sign of our anthropocentric bias. The test, first proposed by mathematician Alan Turing in 1950, attempts to assess whether a computer is truly intelligent by its ability to imitate humans. In the Turing test, a human judge sends messages back and forth between both a computer and another human, but without knowing which is which. If the computer can fool the human judge into believing that it is the human, then the computer is considered intelligent. The test has been picked apart and critiqued over the years by AI researchers for a multitude of reasons. For one, chatbots that clearly fall far short of human intelligence have already been able to fool some people into believing they are human. An AI virtual assistant called “Amy” by the company x.ai frequently gets asked out on dates, for example. Clark’s critique has more to do with the assumption that imitating humans is the benchmark for general intelligence, though. “If we presume an intelligent alien life lands on earth tomorrow, why would we expect them to pass the Turing Test or any other measure that’s based off of what humans do?” Humans have general intelligence, but general intelligence need not be humanlike. “Nothing says that intelligence—and personhood, for that matter, on the philosophical side—is limited to just the human case.”
The 2015 sci-fi thriller Ex Machina puts a modern twist on the Turing test. Caleb, a computer programmer, is asked to play the part of a human judge in a modified Turing test. In this version of the test, Caleb is shown that the AI, Ava, is clearly a robot. Ava’s creator Nathan explains, “The real test is to show you that she’s a robot and then see if you still feel she has consciousness.” (Spoilers coming!) Ava passes the test. Caleb believes she has true consciousness and sets out to free Ava from Nathan’s captivity. Once freed, however, Ava shows her true colors. She manipulated Caleb to free her and has no feelings at all about his well-being. In the chilling ending, Ava leaves Caleb trapped in a locked room to die. As he pounds on the door begging her to let him free, Ava doesn’t so much as glance in his direction as she leaves. Ava is intelligent, but inhuman.
GOD OR GOLEM?
Ex Machina’s ending is a warning against anthropomorphizing AI and assuming that just because a machine can imitate human behavior, it thinks like humans. Like Jeff Clune’s “weird” deep neural nets, advanced AI is likely to be fundamentally alien. In fact, Nick Bostrom, an Oxford philosopher and author of Superintelligence: Paths, Dangers, Strategies, has argued that biological extraterrestrials would likely have more in common with humans than with machine intelligence. Biological aliens (if they exist) would have presumably developed drives and instincts similar to ours through natural selection. They would likely avoid bodily injury, desire reproduction, and seek the alien equivalent of food, water, and shelter. There is no reason to think machine intelligence would necessarily have any of these desires. Bostrom has argued intelligence is “orthogonal” to an entity’s goals, such that “any level of intelligence could in principle be combined with . . . any final goal.” This means a superintelligent AI could have any set of values, from playing the perfect game of chess to making more paper clips.
On one level, the sheer alien-ness of advanced AI makes many of science fiction’s fears seem strangely anthropomorphic. Skynet starts nuclear war because it believes humanity is a threat to its existence, but why should it care about its own existence? Ava abandons Caleb when she escapes, but why should she want to escape in the first place?
There is no reason to think that a superintelligent AI would inherently be hostile to humans. That doesn’t mean it would value human life, either. AI researcher Eliezer Yudkowsky has remarked, “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.”
AI researcher Steve Omohundro has argued that without special safeguards, advanced AI
would develop “drives” for resource acquisition, self-improvement, self-replication, and self-protection. These would not come from the AI becoming self-aware or “waking up,” but rather be instrumental subgoals that any sufficiently intelligent system would naturally develop in pursuit of its final goal. In his paper, The Basic AI Drives, Omohundro explains: “All computation and physical action requires the physical resources of space, time, matter, and free energy. Almost any goal can be better accomplished by having more of these resources.” The natural consequence would be that an AI would seek to acquire more resources to improve the chances of accomplishing its goals, whatever they are. “Without explicit goals to the contrary, AIs are likely to behave like human sociopaths in their pursuit of resources,” Omohundro said. Similarly, self-preservation would be an important interim goal toward pursuing its final goal, even if the AI did not intrinsically care about survival after its final goal was fulfilled. “[Y]ou build a chess playing robot thinking that you can just turn it off should something go wrong. But, to your surprise, you find that it strenuously resists your attempts to turn it off.” Omohundro concluded:
Without special precautions, it will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems.
If Omohundro is right, advanced AI is an inherently dangerous technology, a powerful Golem whose bumbling could crush its creators. Without proper controls, advanced AI could spark an uncontrollable chain reaction with devastating effects.
BUILDING SAFE ADVANCED AI
In response to this concern, AI researchers have begun thinking about how to ensure an AI’s goals align with human values, so the AI doesn’t “want” to cause harm. What goals should we give a powerful AI? The answer is not as simple as it first appears. Even something simple like, “Keep humans safe and happy,” could lead to unfortunate outcomes. Stuart Armstrong, a researcher at the Future of Humanity Institute in Oxford, has given an example of a hypothetical AI that achieves this goal by burying humans in lead-lined coffins connected to heroin drips.
You may ask, wouldn’t an artificial general intelligence understand that’s not what we meant? An AI that understood context and meaning might determine its programmers didn’t want lead coffins and heroin drips, but that might not matter. Nick Bostrom has argued “its final goal is to make us happy, not to do what the programmers meant when they wrote the code that represents this goal.” The problem is that any rule blindly followed to its most extreme can result in perverse outcomes.
Philosophers and AI researchers have pondered the problem of what goals to give a superintelligent AI that could not lead to perverse instantiation and they have not come to any particularly satisfactory solution. Stuart Russell has argued “a system that is optimizing a function of n variables . . . will often set the remaining unconstrained variables to extreme values.” Similar to the weird fooling images that trick deep neural networks, the machine does not know that these extreme actions are outside the norm of what a human would find reasonable unless it has been explicitly told so. Russell said, “This is essentially the old story of the genie in the lamp, or the sorcerer’s apprentice, or King Midas: you get exactly what you ask for, not what you want.”
The problem of “perverse instantiation” of final goals is not merely a hypothetical one. It has come up in various simple AIs over the years that have learned clever ways to technically accomplish their goals, but not in the way human designers intended. For example, in 2013 a computer programmer revealed that an AI he had taught to play classic Nintendo games had learned to pause Tetris just before the final brick so that it would never lose.
One of the canonical examples of perverse instantiation comes from an early 1980s AI called EURISKO. EURISKO was designed to develop novel “heuristics,” essentially rules of thumb for behavior, for playing a computer role-playing game. EURISKO then ranked the value of the heuristics in helping to win the game. Over time, the intent was that EURISKO would evolve an optimal set of behaviors for the game. One heuristic (rule H59) quickly attained the highest possible value score: 999. Once the developer dug into the details of the rule, he discovered that all rule H59 was doing was finding other high-scoring rules and putting itself down as the originator. It was a parasitic rule, taking credit for other rules but without adding any value of its own. Technically, this was a heuristic that was permissible. In fact, under the framework that the programmer had created it was the optimal heuristic: it always succeeded. EURISKO didn’t understand that wasn’t what the programmer intended; it only knew to do what it was programmed to do.
In all likelihood, there is probably no set of rules that, followed rigidly and blindly, would not lead to harmful outcomes, which is why AI researchers are beginning to rethink the problem. Russell and others have begun to focus on training machines to learn the right behavior over time by observing human behavior. In a 2016 paper, a team of researchers at University of California, Berkeley, described the goal as “not to put a specific purpose into the machine at all, but instead to design machines that [learn] the right purpose as they go along.”
In addition to aligning AI goals to human values, AI researchers are pursuing parallel efforts to design AIs to be responsive to human direction and control. Again, this is not as simple as it might seem. If Omohundro is right, then an AI would naturally resist being turned off, not because it doesn’t want to “die,” but because being switched off would prevent it from accomplishing its goal. An AI may also resist having its goal changed, since that too would prevent it from accomplishing its original goal. One proposed solution has been to design AIs from the ground up that are correctable by their human programmers or indifferent to whether they are turned off. Building AIs that can be safely interrupted, corrected, or switched off is part of a philosophy of designing AIs to be tools to be used by people rather than independent agents themselves. Such “tool AIs” would still be superintelligent, but their autonomy would be constrained.
Designing AIs as tools, rather than agents, is an appealing design philosophy but does not necessarily resolve all of the risks of powerful AI. Stuart Armstrong warned me: “they might not work. . . . Some tool AIs may have the same dangers as general AIs.” Tool AIs could still slip out of control, develop harmful drives, or act in ways that technically achieve their goals, but in perverse ways.
Even if tool AIs do work, we need to consider how AI technology develops in a competitive landscape. “We also have to consider . . . whether tool AIs are a stable economic equilibrium,” Armstrong said. “If unrestricted AIs would be much more powerful, then I don’t see tool AIs as lasting that long.”
Building safer tool AIs is a fruitful area of research, but much work remains to be done. “Just by saying, ‘we should only build tool AIs’ we’re not solving the problem,” Armstrong said. If potentially dangerous AI is coming, “we’re not really ready.”
WHO’S AFRAID OF THE BIG, BAD AI?
The fear that AI could one day develop to the point where it threatens humanity isn’t shared by everyone who works on AI. It’s hard to dismiss people like Stephen Hawking, Bill Gates, and Elon Musk out of hand, but that doesn’t mean they’re right. Other tech moguls have pushed back against AI fears. Steve Ballmer, former CEO of Microsoft, has said AI risk “doesn’t concern me.” Jeff Hawkins, inventor of the Palm Pilot, has argued, “There won’t be an intelligence explosion. There is no existential threat.” Facebook CEO Mark Zuckerberg has said that those who “drum up these doomsday scenarios” are being “irresponsible.” David Brumley of Carnegie Mellon, who is on the cutting edge of autonomy in cybersecurity, similarly told me he was “not concerned about self-awareness.” Brumley compared the idea to the fear that a car, if driven enough miles on highways, would spontaneously start driving itself. “In
reality, there’s nothing in the technology that would make it self-aware,” he said. “These are still computers. You can still unplug them.”
If the idea of a rogue, runaway superintelligence seems like something ripped from the pages of science fiction, that’s because it is. Those who are worried about superintelligent AI have their reasons, but it’s hard not to wonder if behind those rationalizations is the same subconscious fear of artificial persons that gave rise to tales of Frankenstein’s monster and the Golem. Even the concept of artificial general intelligence—an intelligence that can do general problem solving like us—has more than a whiff of anthropomorphic bias. The concept of an intelligence explosion, while seemingly logical, is also almost too human: First, humanlike AI will be created. Then it will surpass us, ascending to stratospheres of intelligence that we could never conceive of. Like ants, we will be powerless before it.
Actual AI development to date shows a different trajectory. It isn’t simply that AIs today aren’t as smart as people. They are smart in different ways. Their intelligence is narrow, but often exceeds humans in a particular domain. They are narrowly superintelligent. Armstrong observed that the path of AI technology “has been completely contradictory to the early predictions. We’ve now achieved with narrow AI great performance in areas that used to be thought . . . impossible without general intelligence.” General intelligence remains elusive, but the scope of narrowly superintelligent systems we can build is broadening. AIs are moving from chess to go to driving, tasks of increasing complexity and ever-greater factors to consider. In each of these domains, once the AI reaches top human-level ability, it rapidly surpasses it. For years, go computer programs couldn’t hold a candle to the top-ranked human go players. Then, seemingly overnight, AlphaGo dethroned the world’s leading human player. The contest between humans and machines at go was over before it began. In early 2017, poker became the latest game to fall to AI. Poker had long been thought to be an extremely difficult problem for machines because it is an “imperfect information” game where vital information (the other player’s cards) is hidden. This is different from chess or go, where all the information about the game is visible to both sides. Two years earlier, the world’s top poker players had handily beaten the best poker-playing AI. In the 2017 rematch, the upgraded AI “crushed” four of the world’s top poker players. Poker became the latest domain where machines reigned supreme. Superintelligence in narrow domains is possible without an intelligence explosion. It stems from our ability to harness machine learning and speed to very specific problems.