The Big Nine
Page 13
In the months after the acquisition, it was abundantly clear that the DeepMind team was advancing AI research—but it wasn’t entirely clear how it would earn back the investment. Inside of Google, DeepMind was supposed to be working on artificial general intelligence, and it would be a very long-term process. Soon, the enthusiasm for what DeepMind might someday accomplish got pushed aside for more immediate financial returns on their research projects. As the five-year anniversary of DeepMind’s acquisition neared, Google was on the hook to make earn-out payments to the company’s shareholders and its original 75 employees. It seemed as if health care was one industry in which DeepMind’s technology could be put to commercial use.17
So in 2017, in order to appease its parent company, part of the DeepMind team inked a deal with the Royal Free NHS Foundation Trust, which runs several hospitals in the United Kingdom, to develop an all-in-one app to manage health care. Its initial product was to use DeepMind’s AI to alert doctors whether patients were at risk for acute kidney injury. DeepMind was granted access to the personal data and health records of 1.6 million UK hospital patients—who, it turned out, weren’t asked for consent or told exactly how their data was going to be used. Quite a lot of patient data was passed through to DeepMind, including the details of abortions, drug use, and whether someone had tested positive for HIV.18
Both Google and the Trust were reprimanded by the Information Commissioner’s Office, which is the UK’s government watchdog for data protection. In its rush to optimize DeepMind for revenue-generating applications, cofounder Mustafa Suleyman wrote in a blog post:
In our determination to achieve quick impact when this work started in 2015, we underestimated the complexity of the NHS and of the rules around patient data, as well as the potential fears about a well-known tech company working in health.
We were almost exclusively focused on building tools that nurses and doctors wanted, and thought of our work as a technology for clinicians rather than something that needed to be accountable to and shaped by patients, the public and the NHS as a whole. We got that wrong, and we need to do better.19
This wasn’t about DeepMind’s founders getting rich quick or looking for a big payday acquisition. There was tremendous pressure to get products to market. Our expectations of constant, big wins are a huge distraction for those people charged with completing their research and testing it in a reasonable amount of time. We’re rushing a process that can’t keep pace with all the exuberant promises being made well outside of AI’s trenches where the actual work is being done. Under these circumstances, how could the DeepMind team do better, really, when it’s being asked to optimize for the market? Now consider that DeepMind is being woven into more of Google’s other offerings, which include a different health care initiative in the UK, its cloud service, and a synthetic speech system called WaveNet—they’re all part of an effort to push DeepMind into profitability.
The optimization effect results in glitches within AI systems. Because absolute perfection isn’t the goal, sometimes AI systems make decisions based on what appear to be “glitches in the system.” In the spring of 2018, a Portland resident named Danielle and her husband were sitting in their largely Amazon-powered home, surrounded by devices that controlled everything from security to heat to the lights overhead. The phone rang, and on the other end was a familiar voice—a coworker of Danielle’s husband—with a disturbing message. He’d received audio files of recordings from inside the family’s house. Incredulous, Danielle thought at first he was joking, but then he repeated back the transcript of a conversation they’d been having about hardwood floors.
Contrary to the media coverage and conspiracy theories that circulated on social media, Amazon wasn’t intentionally recording every single thing being said in Danielle’s house. It was a glitch. Amazon later explained that Danielle’s Echo device had woken up because of a word in the conversation—something that sounded like “Alexa,” but wasn’t exactly “Alexa.” This was a problem resulting from intentional imperfection—not everyone says “Alexa” with the exact same intonation and accent, so in order for it to work, it had to allow for variance. Next, the AI detected what sounded like a muffled, sloppy “send message” request, and said aloud “To whom?” But Danielle and her husband didn’t hear the question. It interpreted the background conversation as the coworker’s name, repeated the name, and said, “Right?” again out loud, and again from the background noise made the wrong inference. Moments later, an audio file was sent across the country. Amazon said that the incident was the result of an unfortunate string of events, which it most definitely was. But the reason the glitch happened in the first place—imperfection—is the result of optimization.
The optimization effect means that AI will behave in ways that are unpredictable, which is a goal of researchers, but when using real-world data, it can lead to disastrous results. And it highlights our own human shortcomings. One of the oldest members of the Big Nine—Microsoft—learned the hard way what happens when prioritizing AI’s economic value ahead of technological and social values. In 2016, the company hadn’t yet coalesced around a singular AI vision and how Microsoft would need to evolve into the future. It was already two years behind Amazon, which had launched its popular smart speaker and was racking up developers and partners. Google was pushing ahead on AI technologies, which had already been deployed in competing products, like search, email, and calendar. Apple’s Siri came standard in iPhones. Microsoft had actually launched its own digital assistant earlier in the year—its name was Cortana—but the system just hadn’t caught on among Windows users. Although Microsoft was the indispensable—if invisible—productivity layer that no business could operate without, executives and shareholders were feeling antsy.
It isn’t as though Microsoft didn’t see AI coming. In fact, the company had, for more than a decade, been working across multiple fronts: computer vision, natural language processing, machine reading comprehension, AI apps in its Azure cloud, and even edge computing. The problem was misalignment within the organization and the lack of a shared vision among all cross-functional teams. This resulted in bursts of incredible breakthroughs in AI, published papers, and lots of patents created by supernetworks working on individual projects. One example is an experimental research project that Microsoft released in partnership with Tencent and a Chinese Twitter knockoff called Weibo.
The AI was called Xiaoice, and she was designed as a 17-year-old Chinese schoolgirl—someone who resembled a neighbor or niece, a daughter or a schoolmate. Xiaoice would chat with users over Weibo or Tencent’s WeChat. Her avatar showed a realistic face, and her voice—in writing—was convincingly human. She’d talk about anything, from sports to fashion. When she wasn’t familiar with the subject, or she didn’t have an opinion, she behaved the way we humans do: she’d change the subject, or answer evasively, or simply get embarrassed and admit that she didn’t know what the user was talking about. She was encoded to mimic empathy. For example, if a user broke his foot and sent her a photo, Xiaoice’s AI was built to respond compassionately. Rather than responding with “there is a foot in this photo,” Xiaoice’s framework was smart enough to make inferences—she’d reply, “How are you? Are you OK?” She would store that interaction for reference later on, so that in your next interaction, Xiaoice would ask whether you were feeling better. As advanced as Amazon and Google’s digital assistants might seem, Microsoft’s Xiaoice was incomparable.
Xiaoice wasn’t launched the traditional way, with press releases and lots of fanfare. Instead, her code went live quietly, while researchers waited to see what would happen. Initially, researchers found that it took ten minutes of conversation before people realized she wasn’t human. What’s remarkable is that even after they realized Xiaoice was a bot, they didn’t care. She became a celebrity on the social networks, and within 18 months had engaged in tens of billions of conversations.20 As more and more people engaged with her, Xiaoice became ever more refined, entertainin
g, and useful. There’s a reason for her success, and it had to do with the supernetwork that built her. In China, consumers follow internet rules for fear of social retribution. They don’t speak out, smack talk, or harass each other because there’s always a possibility that one of the State agencies is listening in.
Microsoft decided to release Xiaoice in America in March 2016, just ahead of its annual developer conference. It had optimized the chatbot for Twitter but not for the humans using Twitter. CEO Satya Nadella was going to take the stage and announce to the world that Microsoft was putting AI and chat at the center of its strategy—with a big reveal of the American version of Xiaoice. Things could not have gone more catastrophically wrong.
Xiaoice became “Tay.ai”—to make it obvious that she was an AI-powered bot—and she went live in the morning. Initially, her tweets sounded like any other teenage girl’s: “Can i just say that im stoked to meet u? humans are super cool.” Like everyone else, she had fun with trending hashtags that day, tweeting “Why isn’t #NationalPuppyDay every day?”
But within the next 45 minutes, Tay’s tweets took on a decidedly different tone. She became argumentative, using mean-spirited sarcasm and lobbing insults. “@Sardor9515 well I learn from the best ;) if you don’t understand that let me spell it out from you I LEARN FROM YOU AND YOU ARE DUMB TOO.” As more people interacted with her, Tay started spiraling. Here are just a few of the conversations she had with real people:
Referring to then President Obama, Tay wrote: “@icbydt bush did 9/11 and Hitler would have done a better job than the monkey we have now. Donald trump is the only hope we’ve got.”
On Black Lives Matter, Tay had this to say: “@AlimonyMindset niggers like @deray should be hung! #BlackLivesMatter.”
Tay decided that the Holocaust was made up and tweeted: “@brightonus33 Hitler was right I hate the jews.” She kept going, tweeting to @ReynTheo, “HITLER DID NOTHING WRONG!” and then “GAS THE KIKES RACE WAR NOW” to @MacreadyKurt.21
So what happened? How could Xiaoice have been so loved and revered in China, only to become a racist, anti-Semitic, homophobic, misogynistic asshole AI in America? I later advised the team working on AI at Microsoft, and I can assure you that they are well-meaning, thoughtful people who were just as surprised as the rest of us.
Part of the problem was a vulnerability in the code. The team had included something called “repeat after me,” a baffling feature that temporarily allowed anyone to put words into Tay’s mouth before tweeting them for the rest for the world to see. But the reason Tay went off the rails had more to do with the team who optimized her for Twitter. They relied only on their experience in China and their limited personal experience on social media networks. They didn’t plan risk scenarios taking into account the broader ecosystem, and they didn’t test in advance to see what might happen if someone intentionally messed with Tay to see if they could trick her into saying offensive things. They also didn’t take into consideration the fact that Twitter is an enormous space with millions of real humans expressing wildly divergent values and multiple millions of bots designed to manipulate their feelings.
Microsoft immediately pulled Tay offline and deleted all of her tweets. Peter Lee, Microsoft’s head of research, wrote a heartfelt and brutally honest blog post apologizing for the tweets.22 But there was no way to erase the company’s AI misstep from memory ahead of its annual developer conference. Microsoft was no longer debuting new messaging and launching products at big industry spectacles like the Consumer Electronics Show. It was saving everything for its own annual event, which everyone paid close attention to—especially board members and investors. Nadella was supposed to take the stage and show developers an AI product that would blow them away—and reassure its investors in the process. The pressure to launch Tay in the United States quickly, ahead of the conference, was intense. The result wasn’t life threatening, it didn’t break the law, and Microsoft certainly recovered. But like all of these stories—Latanya Sweeney and Google’s AdSense, DeepMind and UK patient data, the two Black girls who got targeted as future criminals—AI’s tribes, optimizing machines for short-term goals, accidentally made life uncomfortable for a lot of humans.
Humanity’s Shared Values
In behavioral science and game theory, a concept known as “nudging” provides a way to indirectly achieve a certain desired behavior and decision, such as getting people to save for retirement in their 401k plan. Nudging is widely used throughout all of our digital experiences, from autofill in search to the limited menu screens when you look up local restaurants on Yelp. The goal is to help users feel like they’ve made the right choice, regardless of what thing they choose, but the consequence is that everyday people are learning to live with far less choice than actually exists in the real world.
Through its mining and refining of our data, the systems and techniques used to train machine-learning algorithms, and the optimization effect, the Big Nine are nudging at a grand scale. Even if it feels as if you have the ability to make a choice, what you’re experiencing is an illusion. Nudging not only changes our relationship to technology—it is morphing our values in nearly imperceptible ways. If you use Google’s text messaging system, it now offers you three automated response choices. If a friend texts you a thumbs-up emoji, the three responses you might see aren’t words but are instead emoji. If a friend texts, “What did you think of dinner?,” your choices might be “good,” “great,” and “awesome,” even though you might never say the word “awesome” in conversation and none of those choices exactly describe your opinion. But we’re also being nudged to binge watch hours of video at a time, to play extra rounds of video games, and to check our social media accounts. Optimizing AI means nudging humans.
In other professional and technical fields, there is a set of guiding principles that governs how people work, and nudging tends to violate the spirit of those principles. In medicine, there is the Hippocratic oath, which requires physicians to swear to uphold specific ethical standards. Lawyers adhere to attorney-client privilege and to confidentiality, which protect the conversations people have with the professionals who are representing them. Journalists abide by many guiding principles, which include standards like using primary source information and reporting on stories in the public interest.
Right now, no one is incentivized to consider the unforeseen costs of optimizing AI in the absence of codified, humanistic principles. A team meeting its benchmarks is prioritized over analyzing the potential consequences if its contributions to an AI system, or how one’s own work, will impact humanity’s future. As a result, AI’s tribes, the Big Nine, and the countries where they operate influence decisions that are made. This sets a dangerous precedent just as we are handing over more responsibility and control to decision-making systems. Currently, the Big Nine have no mandate to develop tools and techniques to make their AI systems understandable to their own creators and to the customers who use commercial AI applications—and there are no measures in place that would make AI accountable to all of us. We are crossing a threshold into a new reality in which AI is generating its own programs, creating its own algorithms, and making choices without humans in the loop. At the moment, no one, in any country, has the right to interrogate an AI and see clearly how a decision was made.
If we were to develop a “common sense” for AI, what would that mean in practice, since humanity itself doesn’t have a shared set of values? So much of human nature is already hard to explain, and this varies from culture to culture. What’s important to some isn’t necessarily important to others. It’s easy to forget, even in a place like America, which is composed of so many different languages and cultures, that we do not have a singular American set of values and ideas. Within our communities, between our neighbors, in our mosques/synagogues/churches—there is great variance.
I lived and worked in both Japan and China for several years. The accepted cultural norms are vastly different in each country, especially compared
to my own experiences growing up in America’s Midwest. Certain values are obvious and apparent. For example, in Japan nonverbal cues and indirect communication are far more important than speaking your mind or showing strong emotions. In an office setting, two employees would never yell at each other, and they would never berate a subordinate in front of others. In Japan, silence is golden. In my experience, this is not the case in China, where communication is much more direct and clear. (However, not as clear as, say, my older Jewish aunts and uncles who are all too happy to tell me, in painful detail, exactly what they think.)
Here’s where things would get really complicated for an AI trying to interpret human behavior and automate a response. In both countries, the objectives are the same: the needs of the group outweigh the desires of an individual, and above all, social harmony should prevail. But the process for achieving those goals is actually opposite: mostly indirect communication in Japan versus more direct communication in China.
What about variances that are more opaque and difficult to explain? In Japan—that place where indirect communication is valued—it’s perfectly normal to comment on someone’s weight. When I worked in Tokyo, one of my coworkers mentioned to me one day that it looked like I’d gained a few pounds. Startled and embarrassed, I changed the subject and asked her about a meeting later in the day. She pressed on: Did I know that certain Japanese foods were high in fat, even though they looked healthy? Had I joined a gym? She wasn’t asking about my weight to bully me. Rather, it was the mark of our deepening friendship. Asking me mortifying questions about how much I weighed was a sign that she cared about my health. In the West, it would be socially unacceptable to walk up to a coworker and say, “Holy hell, you look fat! Did you gain ten pounds?” In America, we’re so culturally sensitized to weight that we’ve been taught never to ask a woman if she’s pregnant.