Machines of Loving Grace

by John Markoff

  In 1999, Rubin started Palo Alto–based Danger, Inc., a smartphone handset maker, with two close friends who had also been Apple engineers. The company name reflected Rubin’s early obsession with robots. (In the 1960s science-fiction television series Lost in Space, a robot guardian for a young boy would say “Danger, Will Robinson!” whenever trouble loomed.) Danger created an early smartphone called the Sidekick, which was released in 2002. It attracted a diverse cult following with its switchblade-style slide-out keyboard, downloadable software, email, and backups of personal information in “the cloud.” While most businesspeople were still chained to their BlackBerrys, the Sidekick found popularity among young people and hipsters, many of whom switched from PalmPilots.

  Rubin was a member of a unique “Band of Brothers” who passed through Apple Computer in the 1980s, a generation of young computer engineers who came of age in Silicon Valley as disciples of Steve Jobs. Captivated by Jobs’s charisma and his dedication to using good design and computing technology as levers to “change the world,” they set out independently on their own technology quests. The Band of Brothers reflected the tremendous influence Jobs’s Macintosh project had on an entire Silicon Valley generation, and many stayed friends for years afterward. Silicon Valley’s best and brightest believed deeply in bringing the Next Big Thing to millions of people.

  Rubin’s robot obsession, however, was extraordinary, even by the standards of his technology-obsessed engineering friends. While working on phones at Google, he bought an $80,000 robot arm and brought it to work, determined to program it to make espresso—a project that stalled for more than a year because one step in the process required more strength than the arm could exert.

  Early on, Rubin had acquired the Internet domain name android.com, and friends would teasingly even refer to him as “the android.” In his home in the hills near Palo Alto, evidence of the coming world of robots was everywhere, because, once again, Andy Rubin had seen something that hadn’t yet dawned on most others in Silicon Valley. Rubin would soon get the opportunity to make the case for the coming age of mobile robots on a much larger stage.

  In the spring of 2013, Google CEO Larry Page received a curious email. Seated in his office at the company’s Mountain View headquarters, he read a message that warned him an alien attack was under way. Immediately after he read the message, two large men burst into his office and instructed him that it was essential he immediately accompany them to an undisclosed location in Woodside, the elite community populated by Silicon Valley’s technology executives and venture capitalists.

  This was Page’s surprise fortieth birthday party, orchestrated by his wife, Lucy Southworth, a Stanford bioinformatics Ph.D. A crowd of 150 people in appropriate alien-themed costumes had gathered, including Google cofounder Sergey Brin, who wore a dress. In the basement of the sprawling mansion where the party was held, a robot arm grabbed small boxes one at a time and gaily tossed the souvenirs to an appreciative crowd. The robot itself consisted of a standard Japanese-made industrial robot arm outfitted with a suction gripper hand driven by a noisy air compressor. It helped that the robot could “see” the party favors it was picking up. For eyes—actually a single “eye”—the robot used the same sensor Microsoft originally added to the Xbox to capture the gestures of video game players in the living room.

  The box-throwing robot was a prototype designed by Industrial Perception, Inc., a small team then located in a garage just across the freeway from the Googleplex in Palo Alto. When the robot, which had already briefly become an Internet sensation after a video showing its box-tossing antics had appeared on YouTube,10 wasn’t slinging boxes, it was being prototyped as a new class of intelligent industrial labor that might take over tasks as diverse as loading and unloading trucks, packing in warehouses, working on assembly lines, and restocking grocery shelves.

  Equipping the robots to understand what they are seeing was only part of the challenge. Recognizing six-sided boxes had proven not to be an insurmountable problem, although AI researchers only recently solved it. Identifying wanted items on grocery shelves, for example, is an immensely more complicated challenge, and today it still exceeds the capability of the best robot programmers. However, at the Page party, the Yaskawa robot had no apparent difficulty finding the party favor boxes, each of which contained a commemorative T-shirt. Ironically, humans had packed each of those boxes, because the robot was not yet able to handle loose shirts.

  The Industrial Perception arm wasn’t the only intelligent machine at the party. A telepresence robot was out on the dance floor, swaying to the music. It was midnight in Woodside, but Dean Kamen, the inventor of the Segway, was controlling the robot from New Hampshire—where it was now three A.M.

  This robot, dubbed a “Beam,” was from Suitable Technologies, another small start-up just a couple of blocks away from Industrial Perception. Both companies were spin-offs from Willow Garage, a robotics laboratory funded by Scott Hassan, a Stanford graduate school classmate and friend of Page’s. Hassan had been the original programmer of the Google search engine while it was still a Stanford research project. Willow Garage was his effort to build a humanoid robot as a research platform. The company had developed a freely available operating system for robotics as well as a humanoid telepresence robot, PR2, that was being used in a number of universities.

  That evening, both AI and IA technologies were thus in attendance at Page’s party—one of the robots attempted to replace humans while another attempted to augment them. Later that year Google acquired Industrial Perception, the box-handling company, for Rubin’s new robot empire.

  Scott Hassan’s Willow Garage spin-offs once again pose the “end of work” question. Are Page and Hassan architects of a generation of technology that will deeply disrupt the economy by displacing both white-collar and blue-collar workers? Viewed as a one-to-one replacement for humans, the Industrial Perception box handler, which will load or unload a truck, is a significant step into what has been one of the last bastions of unskilled human labor. Warehouse workers, longshoremen, and lumpers all have rough jobs that are low paying and unrewarding. Human workers moving boxes—which can weigh fifty pounds or more—roughly every six seconds get tired and often hurt their backs and wind up disabled.

  The Industrial Perception engineers determined that to win contracts in warehouse and logistics operations, they needed to demonstrate that their robots could reliably move boxes at four-second intervals. Even before their acquisition by Google, they were very close to that goal. However, from the point of view of American workers, a very different picture emerges. In fact, the FedExes, UPSes, Walmarts, and U.S. Post Offices that now employ many of the nation’s unskilled laborers are no longer primarily worried about labor costs and are not anxious to displace workers with lower-cost machines. Many of the workers, it turns out, have already been displaced. The companies are instead faced with an aging workforce and the reality of a labor scarcity. In the very narrow case of loading and unloading trucks, at least, it’s possible the robots have arrived just in time. The deeper and as yet unanswered question remains whether our society will commit to helping its human workers across the new automation divide.

  At the end of 2013 in a nondescript warehouse set behind a furniture store in North Miami, a group of young Japanese engineers began running practice sessions fully a month before the DARPA Robotics Challenge. They had studied under Masayuki Inaba, the well-known roboticist who himself was the prize student of the dean of Japanese robotics, Hirochika Inoue. Inoue had started his work in robotics during graduate school in 1965 when his graduate thesis advisor proposed that he design a mechanical hand to turn a crank.

  Robots have resonated culturally in Japan more positively than they have in the United States. America has long been torn between the robot as a heroic “man of steel” and the image of a Terminator. (Of course, one might reasonably wonder what Americans really felt about the Terminator after Californians twice elected the Hollywood actor who portr
ayed it as governor!) In Japan, however, during the 1950s and 1960s the cartoon robot character Mighty Atom, called Astro Boy in other countries, had framed the more universally positive view of robotics. To some extent, this makes sense: Japan is an aging society, and the Japanese believe they will need autonomous machines to care for their elderly.

  The Japanese team, which named themselves Schaft, came out of JSK, the laboratory Dr. Inoue had established at Tokyo University, early in 2013 with the aim of entering the DARPA Robotics Challenge. They had been forced to spin off from Tokyo University because the school, influenced by the antimilitarist period after the end of World War II, prevented the university laboratory from participating in any event that was sponsored by the U.S. military.11 The team took its name from a 1990s Japanese musical group of the electro-industrial rock genre. Rubin had found the researchers through Marc Raibert.

  When news broke that Google had acquired Schaft, it touched off a good deal of hand-wringing in Japan. There was great pride in the country’s robotics technology. Not only had the Japanese excelled at building walking machines, but for years they had commercialized some of the most sophisticated robots, even as consumer products. Sony had introduced Aibo, a robotic pet dog, in 1999, and continued to offer improved versions until 2005. Following Aibo, a two-foot-tall robot, Qrio, was developed and marketed but never sold commercially. Now it appeared that Google was waltzing in and skimming the cream from decades of Japanese research.

  The reality, however, is that while the Japanese dominated the first-generation robot arms, other nations are now rapidly catching up. Most of the software-centric next-generation robot development work and related artificial intelligence research was happening in the United States. Both Silicon Valley and Route 128 around Boston had once again become hotbeds of robotics start-up activity in 2012 and 2013.

  When they agreed to join Rubin’s expanding robot empire, the Schaft researchers felt conflicted. They expected that now that they were marching to Google’s drumbeat, they would have to give up their dream of competing in the Pentagon contest. “No way!” Rubin told them. “Of course you’re going to stay in the competition.” The ink was barely dry on the Google contract when the Japanese engineers threw themselves into the contest. They started building three prototype machines immediately and they built mockups of each of the eight contest tasks—rough terrain, a door to open, a valve to close, a ladder, and so on—so they could start testing their robots immediately. In June, when DARPA officials checked on the progress of each group, Team Schaft’s thorough preparation stunned Gill Pratt—at the time, none of the other teams had even started!

  In September, when two members of the Schaft team traveled to a DARPA evaluation meeting held in Atlanta alongside the Humanoids 2013 technical conference, they brought a video to demonstrate their progress. Though they spoke almost no English, the video hit like a thunderbolt. The video showed that the young Japanese had solved all the programming problems while the other competitors were still learning how to program their robots. The other teams at the conference were visibly in shock when the two young engineers left the stage. Two months later in their Miami warehouse, the team had settled in and recreated a test course made from plywood. Even though it was almost December, muggy and miserable Miami weather and mosquitoes plagued the researchers. A local security guard who watched over the team was so bitten that he ended up in the hospital after a severe allergic reaction.

  Schaft established a control station on a long table in the cavernous building. Controlling the robot was dead simple—users operated the machine with a Sony Playstation PS3 controller, just like a video game. The robot pilot borrowed sound bites from Nintendo games and added his own special audio feedback to the robot. The researchers practiced each of the tasks over and over until the robot could maneuver the course perfectly.

  Homestead-Miami Speedway was no stranger to growling machines. When it hosts NASCAR races, the stands are usually filled with good ol’ Southern boys. In December of 2013, however, the Robot Challenge had a decidedly different flavor. Raibert called it “Woodstock for robots.” He was there to oversee both the supporting role that Boston Dynamics was playing in technical care and feeding for the Atlas humanoid robots and the splashy demonstrations of several Pentagon-funded four-legged running and walking robots. These machines would periodically trot or gallop along the racecourse to the amazement of the audience of several thousand. DARPA also hosted a robot fair with several dozen exhibitors during the two days of robot competition, which generated a modest crowd as well as a fairly hefty media contingent.

  Google underscored the growing impact of robotics on all aspects of society when it publicly announced Rubin’s robotics division just weeks before the Robotics Challenge. At the beginning of that month, 60 Minutes had aired a segment about Jeff Bezos and Amazon that included a scene in which Bezos led Charlie Rose into a laboratory and showed off an octocopter drone designed to deliver Amazon products autonomously “in 30 minutes.”12 The report sparked another flurry of discussions about the growing role of robots in society. The storage and distribution of commercial goods is already a vast business in the United States, and Amazon has quickly become a dominant low-cost competitor. Google is intent on competing against Amazon in the distribution of all kinds of goods, which will create pressure to automate warehouse processes and move distribution points closer to consumers. If the warehouse was close enough to a consumer—within just blocks, for example, in a large city—why not use a drone for the “last mile”? The idea felt like science fiction come to life, and Rose, who appeared stunned, did not ask hard questions.

  Google, however, unveiled its own drone delivery research project. Just days after the Amazon 60 Minutes extravaganza, the New York Times reported on Google’s robotic ambitions, which dwarfed what Bezos had sketched on the TV news show. Rubin had stepped down as head of Google’s Android phone division in the spring of 2013. Despite reports that he had lost a power struggle and was held in disfavor, exactly the opposite was true. Larry Page, Google’s chief executive, had opened the corporate checkbook and sent Rubin on a remarkable shopping spree. Rubin had spent hundreds of millions of dollars recruiting the best robotics talent and buying the best robotic technology in the world. In addition to Schaft, Google had also acquired Industrial Perception, Meka Robotics, and Redwood Robotics, a group of developers of humanoid robots and robot arms in San Francisco led by one of Rodney Brooks’s star students, and Bot & Dolly, a developer of robotic camera systems that had been used to create special effects in the movie Gravity. Boston Dynamics was the exclamation mark in the buying spree.

  Google’s acquisition of an R & D company closely linked to the military instigated a round of speculation. Many suggested that Google, having bought a military robotics firm, might become a weapons maker. Nothing could have been further from the truth. In his discussions with the technologists at the companies he was acquiring, Rubin sketched out a vision of robots that would safely complete tasks performed by delivery workers at UPS and FedEx. If Bezos could dream of delivering consumer goods from the air, then how outlandish would it be for a Google Shopping Express truck to pull up to a home and dispatch a Google robot to your front door? Rubin also had long had a close relationship with Terry Gou, the CEO of Foxconn, the giant Chinese manufacturer. It would not be out of the realm of possibility to supply robots to replace some of Gou’s one million “animals.”

  Google’s timing in unveiling its new robotics effort was perfect. The December 2013 Robotics Challenge was a preliminary trial to be followed by a final event in June of 2015. DARPA organized the first contest into “tracks” broken broadly into teams that supplied their own robots and teams that used the DARPA-supplied Atlas robots from Boston Dynamics. The preliminary trial turned out to be a showcase for Google’s new robot campaign. Rubin and a small entourage flew into an airport north of Miami on one of Google’s G5 corporate jets and were met by two air-conditioned buses rented for the joint ope
ration.

  The contest consisted of the eight separate tasks performed over two days. The Atlas teams had a comparatively short amount of time before the event to program their robots and practice, and it showed. Compared with the nimble four-legged Boston Dynamics demonstration robots, the contestants themselves were slow and painstaking. A further reminder of how little progress had been made was that the robots were tethered from above in order to protect them from damaging falls without hampering their movements.

  The Boston Dynamics Atlas Robot, designed for the DARPA Robot Challenge. Boston Dynamics was later acquired by Google and has designed a second-generation Atlas intended to operate without tether or power connection. (Photo courtesy of DARPA)

  If that wasn’t enough, DARPA gave the teams a little break in the driving task: they allowed human assistants to place the robots in the cars and connect them to the steering wheel and brakes before they drove through a short obstacle course. Even the best teams, including Schaft, drove the course in a stop-and-go fashion, pulling forward a distance and pausing to recalibrate. The slow pace was strikingly reminiscent of the SRI Shakey robot many decades earlier. The robots were not yet autonomous. Their human controllers were hidden away in the garages while the robots performed their tasks on the speedway’s infield, directed via a fiber-optic network that fed video and sensor data back to the operator console workstations. To bedevil the teams and create a real-world sense of crisis, DARPA throttled the data connection at regular intervals. This gave even the best robots a stuttering quality, and the assembled press hunted for metaphors less trite than “watching grass grow” or “watching paint dry” to describe the scene.