Machines of Loving Grace

by John Markoff

  The Sloan Foundation had sent Searle to Yale to discuss the subject of artificial intelligence. While on the plane to the meeting he began reading a book about artificial intelligence written by Roger Schank and Robert Abelson, the leading Yale AI researchers during the second half of the 1970s. Scripts, Plans, Goals, and Understanding16 made the assertion that artificial intelligence programs could “understand” stories that had been designed by their developers. For example, developers could present the computer with a simple story, such as a description of a man going into a restaurant, ordering a hamburger, and then storming out without paying for it. In response to a query, the program was able to infer that the man had not eaten the hamburger. “That can’t be right,” Searle thought to himself, “because you could give me a story in Chinese with a whole lot of rules for shuffling the Chinese symbols, and I don’t understand a word of Chinese but all the same I could give the right answer.”17 He decided that it just didn’t follow that the computer had the ability to understand anything just because it could interpret a set of rules.

  While flying to his lecture, he came up with what has been called the “Chinese Room” argument against sentient machines. Searle’s critique was that there could be no simulated “brains in a box.” His argument was different from the original Dreyfus critique, which asserted that obtaining human-level performance from AI software was impossible. Searle simply argued that a computing machine is little more than a very fast symbol shuffler that uses a set of syntactical rules. What it lacks is what the biological mind has—the ability to interpret semantics. The biological origin of semantics, the formal study of meaning, remains a great mystery. Searle’s argument was infuriating to the AI community in part because he implied that their argument implicitly linked them with a theological argument that the mind is outside the physical, biological world. His argument was that mental processes are entirely caused by biological processes in the brain and they are realized there, and if you want to make a machine that can think, you must duplicate, rather than simulate, those processes. At the time Searle thought that they had probably already considered his objection and the discussion wouldn’t last a week, let alone decades. But it has. Searle’s original article generated thirty published refutations. Three decades later, the debate is anything but settled. To date, there are several hundred published attacks on his idea. And Searle is still alive and busy defending his position.

  It is also notable that the lunchtime discussions about the possibility of intelligent and conceivably self-aware machines took place against a backdrop of the Reagan military buildup. The Vietnam War had ended, but there were still active pockets of political dissent around the country. The philosophers would meet at the Y across the street from the Berkeley campus. Winograd and Danny Bobrow from Xerox PARC had become regular visitors at these lunches, and Winograd found that they challenged his intellectual biases about the philosophical underpinnings of AI.

  He would eventually give up the AI “faith.” Winograd concluded that there was nothing mystical about human intelligence. In principle, if you could discover the way the brain worked, you could build a functional artificially intelligent machine, but you couldn’t build that same machine with symbolic logic and computing, which was the dominant approach in the 1970s and 1980s. Winograd’s interest in artificial intelligence had been twofold: AI served both as a model for understanding language and the human brain and as a system that could perform useful tasks. At that point, however, he took an “Engelbartian” turn. Philosophically and politically, human-centered computing was a better fit with his view of the world. Winograd had gotten intellectually involved with Flores, which led to a book, Understanding Computers and Cognition: A New Foundation for Design, a critique of artificial intelligence. Understanding Computers, though, was philosophy, not science, and Winograd still had to figure out what to do with his career. Eventually, he set down his effort to build smarter machines and focused instead on the question of how to use computers to make people smarter. Winograd crossed the chasm. From designing systems that were intended to supplant humans he turned his focus to working on technologies that enhanced the way people interact with computers.

  Though Winograd would argue years later that politics had not directly played a role in his turn away from artificial intelligence, the political climate of the time certainly influenced many other scientists’ decisions to abandon the artificial intelligence camp. During a crucial period from 1975 to 1985, artificial intelligence research was overwhelmingly funded by the Defense Department. Some of the nation’s most notable computer scientists—including Winograd—had started to worry about the increasing involvement of the military in computing technology R & D. For a generation who had grown up watching the movie Dr. Strangelove, the Reagan administration Star Wars antimissile program seemed like dangerous brinkmanship. It was at least a part of Winograd’s moral background and was clearly part of the intellectual backdrop during the time when he decided to leave the field he had helped to create. Winograd was a self-described “child of the ’60s,”18 and during the crucial years when he turned away from AI, he simultaneously played a key role in building a national organization of computer scientists, led by researchers at Xerox PARC and Stanford, who had become alarmed at the Star Wars weapons buildup. The group shared a deep fear that the U.S. military command would push the country into a nuclear confrontation with the Soviet Union. As a graduate student Winograd had been active against the war in Vietnam while he was in Boston as part of a group called “Computer People for Peace.” In 1981 he became active again as a leader in helping create a national organization of computer scientists who opposed nuclear weapons.

  In response to the highly technical Strategic Defense Initiative, the disaffected computer scientists believed they could use the weight of their expertise to create a more effective anti–nuclear weapons group. They evolved from being “people” and became “professionals.” In 1981, they founded a new organization called Computer Professionals for Social Responsibility. Winograd ran the first planning meeting, held in a large classroom at Stanford. Those who attended recalled that unlike many political meetings from the antiwar era that were marked by acrimony and debate, the evening was characterized by an unusual sense of unity and common purpose. Winograd proved an effective political organizer.

  In a 1984 essay on the question of whether computer scientists should accept military funding, Winograd pointed out that he had avoided applying for military funding in the past, but by keeping his decision private, he had ducked what he would come to view as a broader responsibility. He had, of course, received his training in a military-funded laboratory at MIT. Helping establish Computer Professionals for Social Responsibility was the first of a set of events that would eventually lead Winograd to “desert” the AI community and turn his attention from building intelligent machines to augmenting humans.

  Indirectly it was a move that would have a vast impact on the world. Winograd was recognized enough in the artificial intelligence community that, if he had decided to pursue a more typical academic career, he could have built an academic empire based on his research interests. Personally, however, he had no interest in building a large research lab or even supporting postdoctoral researchers. He was passionate about one-to-one interaction with his students.

  One of these was Larry Page, a brash young man with a wide range of ideas for possible dissertation topics. Under Winograd’s guidance Page settled on the idea of downloading the entire Web and improving the way information was organized and discovered. He set about doing this by mining human knowledge, which was embodied in existing Web hyperlinks. In 1998, Winograd and Page joined with Sergey Brin, another Stanford graduate student and a close friend of Page’s, and Brin’s faculty advisor, Rajeev Motwani, an expert in data mining, to coauthor a journal article titled “What Can You Do with a Web in Your Pocket?”19 In the paper, they described the prototype version of the Google search engine.

  Page ha
d been thinking about other more conventional AI research ideas, like self-driving cars. Instead, with Winograd’s encouragement, he would find an ingenious way of mining human behavior and intelligence by exploiting the links created by millions of Web users. He used this information to significantly improve the quality of the results returned by a search engine. This work would be responsible for the most significant “augmentation” tool in human history. In September of that year, Page and Brin left Stanford and founded Google, Inc. with the modest goal of “organizing the world’s knowledge and making it universally useful.”

  By the end of the 1990s Winograd believed that the artificial intelligence and human-computer interaction research communities represented fundamentally different philosophies about how computers and humans should interact. The easy solution, he argued, would be to agree that both camps were equally “right” and to stipulate that there will obviously be problems in the world that could be solved by either approach. This answer, however, would obscure the fact that inherent in these differing approaches are design consequences that play out in the nature of the systems. Adherents of the different philosophies, of course, construct these systems. Winograd had come to believe that the way computerized systems are designed has consequences both in how we understand humans and how technologies are designed for their benefit.

  The AI approach, which Winograd describes as “rationalistic,” views people as machines. Humans are modeled with internal mechanisms very much like digital computers. “The key assumptions of the rationalistic approach are that the essential aspects of thought can be captured in a formal symbolic representation,” he wrote. “Armed with this logic, we can create intelligent programs and we can design systems that optimize human interaction.”20 In opposition to the rational AI approach was the augmentation method that Winograd describes as “design.” That approach is more common in the human-computer interaction community, in which developers focus not on modeling a single human intelligence, but rather on using the relationship between the human and the environment as the starting point for their investigations, be it with humans or an ensemble of machines. Described as “human-centered” design, this school of thought eschews formal planning in favor of an iterative approach to design, encapsulated well in the words of industrial designer and IDEO founder David Kelley: “Enlightened trial and error outperforms the planning of flawless intellect.”21 Pioneered by psychologists and computer scientists like Donald Norman at the University of California at San Diego and Ben Shneiderman at the University of Maryland, human-centered design would become an increasingly popular approach that veered away from the rationalist AI model that was popularized in the 1980s.

  In the wake of the defeats of the AI Winter in the 1980s, in the 1990s, the artificial intelligence community also changed dramatically. It largely abandoned its original formal, rationalist, top-down straitjacket that had been described as GOFAI, or “Good Old-Fashioned Artificial Intelligence,” in favor of statistical and “bottom-up,” or “constructivist,” approaches, such as those pursued by roboticists led by Rod Brooks. Nevertheless, the two communities have remained distant, preoccupied with their contradictory challenges of either replacing or augmenting human skills.

  In breaking with the AI community, Winograd became a member of a group of scientists and engineers who took a step back and rethought the relationship between humans and the smart tools they were building. In doing so, he also reframed the concept of “machine” intelligence. By posing the question of whether humans were actually “thinking machines” in the same manner of the computing machines that the AI researchers were trying to create, he argued that the very question makes us engage—wittingly or not—in an act of projection that tells us more about our concept of human intelligence than it does about the machines we are trying to understand. Winograd came to believe that intelligence is an artifact of our social nature, and that we flatten our humaneness by simplifying and distorting what it is to be human as simulated by a machine.

  While artificial intelligence researchers rarely spoke to the human-centered design researchers, the two groups would occasionally organize confrontational sessions at technical conferences. In the 1990s, Ben Shneiderman was a University of Maryland computer scientist who had become a passionate advocate of the idea of human-centered design through what became known as “direct manipulation.” During the 1980s, with the advent of Apple’s Macintosh and Microsoft’s Windows software systems, direct manipulation had become the dominant style in computer user interfaces. For example, rather than entering commands on a keyboard, users could change the shape of an image displayed on a computer screen by grabbing its edges or corners with a mouse and dragging them.

  Shneiderman was at the top of his game and, during the 1990s, he was a regular consultant at companies like Apple, where he dispensed advice on how to efficiently design computer interfaces. Shneiderman, who considered himself to be an opponent of AI, counted among his influences Marshall McLuhan. During college, after attending a McLuhan lecture at the Ninety-Second Street Y in New York City, he had felt emboldened to pursue his own various interests, which crossed the boundaries between science and the humanities. He went home and printed a business card describing his job title as “General Eclectic” and subtitled it “Progress is not our most important product.”22

  He would come to take pride in the fact that Terry Winograd had moved from the AI camp to the HCI world. Shneiderman sharply disagreed with Winograd’s thesis when he read it in the 1970s and had written a critical chapter about SHRDLU in his 1980 book Software Psychology. Some years later, when Winograd and Flores published Understanding Computers and Cognition, which made the point that computers were unable to “understand” human language, he called Winograd up and told him, “You were my enemy, but I see you’ve changed.” Winograd laughed and told Shneiderman that Software Psychology was required reading in his classes. The two men became good friends.

  In his lectures and writing, Shneiderman didn’t mince words in his attacks on the AI world. He argued not only that the AI technologies would fail, but also that they were poorly designed and ethically compromised because they were not designed to help humans. With great enthusiasm, he argued that autonomous systems raised profound moral issues related to who was responsible for the actions of the systems, issues that weren’t being addressed by computer researchers. This fervor wasn’t new for Shneiderman, who had previously been involved in legendary shouting matches at technical meetings over the wisdom of designing animated human agents like Microsoft Clippy, the Office assistant, and Bob, the ill-received attempts Microsoft made to design more “friendly” user interfaces.

  In the early 1990s anthropomorphic interfaces had become something of a fad in computer design circles. Inspired in part by Apple’s widely viewed Knowledge Navigator video, computer interface designers were adding helpful and chatty animated cartoon figures to systems. Banks were experimenting with animated characters that would interact with customers from the displays of automated teller machines, and car manufacturers started to design cars with speech synthesis that would, for example, warn drivers when their door was ajar. The initial infatuation would come to an abrupt halt, however, with the embarrassing failure of Microsoft Bob. Although it had been designed with the aid of Stanford University user interface specialists, the program was widely derided as a goofy idea.

  Did the problem with Microsoft Bob lie with the idea of a “social” interface itself, or instead with the way it was implemented? Microsoft’s bumbling efforts were rooted in the work of Stanford researchers Clifford Nass and Byron Reeves, who had discovered that humans responded well to computer interfaces that offered the illusion of human interaction. The two researchers arrived at the Stanford Communications Department simultaneously in 1986. Reeves had been a professor of communications at the University of Wisconsin, and Nass had studied mathematics at Princeton and worked at IBM and Intel before turning his interests toward sociology.


  As a social scientist Nass worked with Reeves to conduct a series of experiments that led to a theory of communications they described as “the Media Equation.” In their book, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places, they explored what they saw as the human desire to interact with technological devices—computers, televisions, and other electronic media—in the same “social” fashion with which they interacted with other humans. After writing The Media Equation, Reeves and Nass were hired as consultants for Microsoft in 1992 and encouraged the design of familiar social and natural interfaces. This extended the thinking underlying Apple’s graphical interface for the Macintosh, which, like Windows, had been inspired by the original work done on the Alto at Xerox PARC. Both were designs that attempted to ease the task of using a computer by creating a graphical environment that was evocative of a desk and office environment in the physical world. However, Microsoft Bob, which attempted to extend the “desktop” metaphor by creating a graphical computer environment that evoked the family home, adopted a cartoonish and dumbed-down approach that the computer digerati found insulting to users, and the customer base overwhelmingly rejected it.

  Decades later the success of Apple’s Siri has vindicated Nass and Reeves’s early research, suggesting that the failure of Microsoft Bob lay in how Microsoft built and applied the system rather than in the approach itself. Siri speeds people up in contexts where keyboard input might be difficult or unsafe, such as while walking or driving. Both Microsoft Bob and Clippy, on the other hand, slowed down user engagement with the program and came across as overly simplistic and condescending to users: “as if they were being asked to learn to ride a bicycle by starting with a tricycle,” according to Tandy Trower, a veteran Microsoft executive.23 That said, Trower pointed out that Microsoft may have fundamentally mistaken the insights offered by the Stanford social scientists: “Nass and Reeves’ research suggests that user expectations of human-like behavior are raised as characters become more human,” he wrote. “This Einstein character sneezed when you asked it to exit. While no users were ever sprayed upon by the character’s departure, if you study Nass and Reeves, this is considered to be socially inappropriate and rude behavior. It doesn’t matter that they are just silly little animations on the screen; most people still respond negatively to such behavior.”24