Machines of Loving Grace

by John Markoff

  Cheyer walked the halls and sampled the different projects at the laboratory, like natural language understanding, speech recognition, cooperating robots, and machine vision. SRI was his playground and he used it to mash together a remarkably disparate and rich set of computing systems and services—and he did it all before he saw his first Web browser. The World Wide Web was just beginning to filter out into the world. When the NCSA Mosaic browser, the first popular browser that brought the Web to the general public, finally did arrive, it felt like déjà vu.

  Cheyer wanted to create an assistant that could provide a computer user with the kind of help he or she might expect to get from an attentive secretary. Although he had started on his own, over the next six years he worked with a small team of programmers and designers and created more than four dozen applications, ranging from intelligent refrigerators that would find recipes and restock themselves to televisions that let you control your home, collaborative robots, and intelligent offices. Ultimately the team would have a significant impact on mobile computing. Fifteen years later, two members of his early research group were key technology executives overseeing the design of the Samsung Galaxy smartphone and three had gone on to Apple to deliver Siri.

  Cheyer quietly earned a reputation inside SRI as the “next Engelbart.” Eventually he became so passionate about Engelbart’s ideas that he kept a photo of the legendary computer scientist on his desk to remind him of his principles. By the end of the 1990s Cheyer was ready for a new challenge. The dot-com era was in full swing and he decided to commercialize his ideas. The business-to-business Internet was exploding and everywhere there were services that needed to be interconnected. His research was a perfect fit for the newly popular idea of loosely coupled control. In a world of networked computers, software that allowed them to cooperate was just beginning to be designed. He was following a similar path to Marty Tenenbaum’s, the AI researcher who had created CommerceNet, the company for which Tom Gruber built ontologies.

  One of a small group of Silicon Valley researchers who realized early on that the Internet would become the glue that connected all commerce, Cheyer went to a competitor called VerticalNet, where he created a research lab and was soon made VP of engineering. Like Gruber, he was caught up in the dot-com maelstrom. At one point VerticalNet’s market value soared to $12 billion on revenues of a little more than $112 million. Of course it couldn’t last, and it didn’t. He stayed for four years and then found his way back to SRI.

  DARPA knocked on Cheyer’s door with an offer to head up Tony Tether’s ambitious national CALO effort, which DARPA anticipated would draw on the efforts of AI researchers around the country. Usually DARPA would simultaneously fund many research labs and not integrate the results. The new DARPA program, however, called for SRI to marshal all the research into the development of CALO. Everyone would report to the SRI team and develop a single integrated system. Cheyer helped write the initial DARPA proposal, and when SRI received the award, he became engineering architect for the project. CALO was rooted firmly in the traditional world of first-generation symbolic artificial intelligence—planning and reasoning and ontologies—but there was also a new focus on what has been described as “learning in the wild.”

  CALO had the trappings of a small Manhattan Project. Over four hundred people were involved at the peak, and the project would generate more than six hundred research papers. DARPA spent almost a quarter billion dollars on the effort, making it one of the most expensive artificial intelligence projects in history. Researchers on the CALO project tried to build a software assistant that would possess humanlike adaptability, learn from the person it worked with, and change its behavior accordingly.

  When CALO passed its annual system tests, DARPA was enthusiastic. Tether awarded the project an excellence prize, and some of the technology made the transition into navy projects. But Adam Cheyer, as engineering architect, had experienced more than his share of frustrations. John McCarthy had famously asserted that building a “thinking machine” would require “1.8 Einsteins and one-tenth the resources of the Manhattan Project.” To put his estimate in perspective, since the Manhattan Project would cost more than $25 billion in current dollars, McCarthy’s estimate would mean that CALO was funded with less than one-tenth of what would be needed to build a thinking machine.

  For Cheyer, however, the principal obstacle in designing CALO was not lack of funding. Rather it was that DARPA tried to micromanage his progress. Often unable to pursue its own agenda, the rest of the management team would shunt aside Cheyer’s ideas. He had a difficult time shepherding the huge number of teams, each of which had its own priorities and received only a small amount of funding from the CALO project. Cheyer’s entreaties to work together on a common project that integrated a huge swath of ideas into a new “cognitive” architecture largely fell on deaf ears. The teams listened politely because they were interested in the next round of money, and they would deliver software, but they all wanted to pursue their own projects. In the end there was no way that a large and bureaucratic program could have a direct impact in the real world.

  To cope with his frustrations he laid out a series of side projects to work on in 2007. They ranged from efforts to commercialize the CALO technology to the creation, with several friends, of an activists’ social network called change.org. It would be a remarkably productive year for Cheyer. With a graduate student, Didier Guzzoni, he used CALO technologies to build a new software development system that eventually became the foundation for Siri. He also put together a small development team that started commercializing various other components of Siri for applications like smartphone calendars and online news reading. He also quietly helped to cofound Genetic Finance, a stealth machine-learning company that built a cluster of more than one million computers to solve financial problems such as predicting the stock market.

  In the midst of all of this, Cheyer approached SRI management to ask for some IR & D funding and told them, “I want a little side project where I’m going to build my own CALO the way it should be done.” He wanted to build a single integrated system, not a patchwork quilt from dozens of different organizations. SRI agreed, and he named his project “Active Ontologies.” He ran it quietly alongside the much larger operation.

  The project gained more traction when a key group of SRI technical leaders met for a daylong retreat in Half Moon Bay, a beach town close to the Menlo Park laboratory. There had been growing pressure to commercialize SRI research from the CEO, Curt Carlson, and CALO was an obvious candidate. The retreat was crucial for hashing out answers to basic questions about the goals for the software, like: What should a personal assistant “feel” like? Should they use an avatar design? Avatars had always been a controversial aspect of the design of virtual assistants. Apple’s Knowledge Navigator video had envisioned a prim young male with a bow tie who looked a bit like Steve Jobs. The CALO project, on the other hand, did not have an avatar. The developers went back and forth on whether the system should be a chatbot, the kind of personal companion that researchers had explored for decades in programs like Eliza that engaged human users in keyboard “conversations.” In the end, they came to a compromise. They decided that nobody was going to sit and chat with a virtual robot all day. Instead, they were going to design a system for people who needed help managing their busy day-to-day lives.

  The group came up with the notion of “delight nuggets.” Because they were trying to create a humanlike persona, they decided to sprinkle endearing phrases into the software. For example, if a user asked the system about the forecast for the day, the system would answer—and if the forecast indicated that it would rain, the system would add: “And don’t forget your umbrella!” The developers wanted to give the user what he or she wanted and to make the design goal about helping them manage their lives—and then to surprise them, just a little bit. Including these phrases added a touch of humanity to the interaction, even though systems did not yet feature speech synthesis and s
peech recognition.

  The 2007 meeting served as a launchpad. SRI’s commercialization board gave the team the approval to begin looking for outside money in August. The name Siri ended up working on a wonderful range of levels. Not only did it mean “secret” in Swahili, but Cheyer had once worked on a project called Iris, which was Siri spelled backward. And of course, everyone liked that the name was also a riff on SRI.

  In 1987 Apple’s chief executive John Sculley gave a keynote address at Educom, the national educational technology conference. He showed a promotional video that a small Apple team had produced to illustrate the idea of something he described as the Knowledge Navigator. At the time, the video, which caught the public’s eye (went “viral” in today’s parlance), seemed impossibly far out. The Knowledge Navigator was a tour de force that pointed the way to a computing world beyond the desktop computer of the mid-1980s. Knowledge Navigator ultimately spawned a seemingly endless stream of high-tech Silicon Valley “vision statements,” including one from Microsoft in 1991 presented by Bill Gates called “Information at Your Fingertips.” Yet at that time, the Knowledge Navigator was early to offer a compelling vision for a future beyond desktop personal computing. The video centered on a conversation between an absentminded professor and a perky, bow-tied on-screen avatar as a guide for both the professor’s research and his day-to-day affairs. It sketched a future in which computer interaction was no longer based on a keyboard and mouse. Instead, the Knowledge Navigator envisioned a natural conversation with an intelligent machine that both recognized and synthesized human speech.

  Brought to Apple as chief executive during the personal computing boom, Sculley started his tenure in 1983 with a well-chronicled romance with Apple’s cofounder Steve Jobs. Later, when the company’s growth stalled in the face of competition from IBM and others, Sculley fought Jobs for control of the company, and won.

  However, in 1986, Jobs launched a new computer company, NeXT. Jobs wanted to make beautiful workstations for college students and faculty researchers. That placed pressure on Sculley to demonstrate that Apple could still innovate without its original visionary. Sculley turned to Alan Kay, who had left Xerox PARC first to create Atari Labs and then came to Apple, for guidance on the future of the computer market. Kay’s conversations with Apple’s chief executive were summarized in a final chapter in Sculley’s autobiographical Odyssey. Kay’s idea centered on “a wonderful fantasy machine called the Knowledge Navigator,”4 which wove together a number of his original Dynabook ideas with concepts that would ultimately take shape in the form of the World Wide Web.

  Alan Kay would later say that John Sculley had asked him to come up with a “modern Dynabook,” which he found humorous, since at the time his original Dynabook still didn’t exist. He said that in response to Sculley’s request, he had pulled together a variety of ideas from his original Dynabook research and the artificial intelligence community, as well as from MIT Media Laboratory director Nicholas Negroponte, an advocate of speech interfaces.5 Negroponte had created the Architecture Machine Group at MIT in 1967, in part inspired by the ideas of Ivan Sutherland, whose “Sketchpad” Ph.D. thesis was a seminal work in both computer graphics and interface design.

  Historians have underestimated Negroponte’s influence on Apple and the computer industry as a whole. Although Negroponte’s “Architecture Machine” idea never gained popular traction, it did have a very specific impact on Bill Atkinson, one of the principal designers of Apple’s Lisa and Macintosh computers. Many of the ideas for Lisa and Macintosh were generated from Negroponte’s early efforts to envision what the field of architecture would be like with the aid of computers. Negroponte’s group created something called “DataLand,” a prototype of a visual data management system. In many ways, DataLand was a much broader exploration of how human computer users might interact with information more fluidly. It was certainly broader in scope than the projects at PARC, which focused more narrowly on a creating a virtual desktop. Indeed, Negroponte’s goal was expansive. DataLand allowed users to view, in a special room, an immersive information environment back-projected on a giant piece of polished glass as a series of thumbnails representing everything from documents to maps. It was like using a Macintosh or a Windows computer, but having the control screen surround you rather than appear on a small display. It was possible to zoom in on and “fly” through the virtual environment by using a joystick, and when you got close to objects like files, they would talk to you (e.g., “This is Nicholas’s Calendar”) in a soothing voice. Atkinson visited Negroponte’s lab and thought this kind of interface could solve Apple’s document filing problem. He wanted to organize documents spatially and place them in proximity to other related documents. Although it was a fascinating concept, it proved unwieldy in practice, and the group returned to something closer to the PARC desktop ideas.

  Kay “channeled” ideas that he had gathered in his discussions with Negroponte, passing them on both to Sculley and to the group that created the Knowledge Navigator video. Kay credited Negroponte with playing what he called the “Wayne Gretzky Game”—skating to where the puck was going, rather than where it was. Kay had eagerly read Gordon Moore’s early Electronics article, which was bold enough to sketch the progress of computing power ten years into the future—1975.6 He drew the line out to 1995 and beyond. This future-oriented approach meant that he could assume that 3-D graphics would be commercially available within just several decades.

  Negroponte represents the “missing link” between Norbert Wiener’s early insights into computing and its consequences, the early world of artificial intelligence, and the explosive rise of the personal computer industry during the 1980s. In the late sixties, Negroponte was teaching a course on computer-aided design for architects at MIT. He was not a great fan of lecturing and so had perfected a Tom Sawyer approach to his course—he brought in many guest lecturers. He attracted a dazzling and diverse array of talent. Isaac Asimov, for example, was living in Cambridge at the time and came to Negroponte’s class to speak each year, as did Gordon Pask, a British cyberneticist who was traveling widely in U.S. computer research circles in the 1960s and 1970s. If Kay was influenced by Negroponte, he in turn would point to the influence and inspiration of Gordon Pask. At the beginning of the interactive computing era Pask had a broad but generally unchronicled influence on computer and cognitive science research in the United States. Ted Nelson met him in the hallways of the University of Illinois Chicago Circle campus and fell under his spell as well. He described Pask affectionately in his Computer Lib manifesto as the “maddest of mad scientists.”

  In 1968, Negroponte, like many in the computing world, was deeply influenced by Ivan Sutherland’s 1963 Ph.D. project, Sketchpad, a graphical and interactive computing tool that pioneered human-computer interaction design. Following in Sutherland’s footsteps, Negroponte began work on an “Architecture Machine” that was intended to help human architects build systems that exceeded their individual intellectual grasp. His first effort to build the machine was a software program called URBAN5. The year after he created it, he took a video of his early Architecture Machine project to the London art exhibition known as Cybernetic Serendipity, which was held at the Institute of Contemporary Arts. The exhibition had assembled a wild variety of mechanical and computerized art exhibits, including large mobiles created by Gordon Pask, designed with interactive parts to permit viewers to enter into a “conversation” with his installations.

  The two met at the exhibition and became good friends. Pask would come to visit the Architecture Machine Group three or four times a year for a week at a time and always stayed as a guest at Marvin Minsky’s home. He was a striking character who dressed the part of an Edwardian dandy, complete with a cape, and who occasionally lapsed into double-talk and wordplay. He was squarely in the Norbert Wiener cybernetics tradition, which had taken hold with more force in Europe than in the United States. Pask was also subtly but significantly at odds with the prevailing art
ificial intelligence world. If AI was about making smart machines that mimicked human capabilities, cybernetics was focused instead on the idea of creating systems to achieve goals.7 Gordon Pask’s insight into the nature of intelligence, which he situated not in the individual but in a conversation between people, strongly influenced Negroponte. Indeed, it was Pask who laid the groundwork for viewing human-machine interactions as conversations that would be later demonstrated by Knowledge Navigator and still later realized in Siri as a conversation: he “conceived human-machine interaction as a form of conversation, a dynamical process, in which the participants learn about each other.”8

  Negroponte was early to grasp Pask’s ideas about computer interaction. Later in the 1970s, Pask’s ideas also influenced Negroponte’s thinking and design at the MIT Media Laboratory as he broadened the original mission of the Architecture Machine Group. Ideas from the Media Lab filtered into Apple’s strategic direction because Kay was close to Negroponte and was spending time teaching there. Few noticed it at the time, but Apple’s release of Siri as a critical addition to the iPhone 4S in October of 2011 fell within two weeks of Kay’s predicted release date for the Knowledge Navigator. The idea traced a direct path from Pask to Negroponte to Kay to the Siri team. A parallel thread ran through Gruber’s original work on computing tools for the disabled, to his work on Intraspect, and to the new project at SRI. In the space of just a generation, a wave of computer-mediated communication technology had inaugurated a new way of facilitating collaboration between humans and machines. Gruber recognized that humans had evolved from using tribal communication to written language, and then quickly to using the telephone and computer communications.

  Computing had become a prosthesis, not in a bad sense, but rather as a way to augment human capabilities as first foreseen by Vannevar Bush, Licklider, and Engelbart. Intraspect and Hypermail had been efforts to build a cognitive prosthesis for work that needed to go beyond the size of a small tribe. The nature of collaboration was changing overnight. People could have conversations when they weren’t in the same room, or even in the same time zone. Simple online email lists like www-talk were being used to develop new Web standards. A permanent archive made it possible for new participants to quickly get up to speed on various issues by reading a record of past conversation.