Machines of Loving Grace

by John Markoff

  Sebastian Thrun had begun building a network of connections in Silicon Valley after he arrived on a sabbatical from CMU several years earlier. One of those was Gary Bradski, an expert in machine vision at Intel Labs in Santa Clara. The company was the world’s largest chipmaker and had developed a manufacturing strategy called “copy exact,” a way of developing next-generation manufacturing techniques to make ever-smaller chips. Intel would develop a new technology at a prototype facility and then export that process to wherever it planned to produce the denser chips in volume. It was a system that required discipline, and Bradski was a bit of a “Wild Duck”—a term that IBM originally used to describe employees who refused to fly in formation—compared to typical engineers in Intel’s regimented semiconductor manufacturing culture.

  A refugee from the high-flying finance world of “quants” on the East Coast, Bradski arrived at Intel in 1996 and was forced to spend a year doing boring grunt work, like developing an image-processing software library for factory automation applications. After paying his dues, he was moved to the chipmaker’s research laboratory and started researching interesting projects. Bradski had grown up in Palo Alto before leaving to study physics and artificial intelligence at Berkeley and Boston University. He returned because he had been bitten by the Silicon Valley entrepreneurial bug.

  For a while he wrote academic research papers about machine vision, but he soon learned that there was no direct payoff. The papers garnered respect at places like Berkeley, Stanford, and MIT, but they didn’t resonate with the rest of Silicon Valley. Besides, he realized that what was special about Intel was its deep pockets. He decided he should be exploiting them. “I should do something that has higher leverage,” he thought to himself.

  In his first year at Intel he met some superstar Russian software designers who worked under contract for the chipmaker, and he realized that they could be an important resource for him. At the time, the open-source software movement was incredibly popular. His background was in computer vision, and so he put two and two together and decided to create a project to build a library of open-source machine vision software tools. Taking the Linux operating system as a reference, it was obvious that when programmers worldwide have access to an extraordinary common set of tools, it makes everybody’s research a lot easier. “I should give everyone that tool in vision research,” he decided.

  While his boss was on sabbatical he launched OpenCV, or Open Source Computer Vision, a software library that made it easier for researchers to develop vision applications using Intel hardware. Bradski was a believer in an iconoclastic operating style that was sometimes attributed to Admiral Grace Hopper and was shared by many who liked getting things done inside large organizations. “Better to seek forgiveness than to ask permission” was his motto. Eventually OpenCV contained a library of more than 2,500 algorithms including both computer vision and machine-learning software. OpenCV also hosted programs that could recognize faces, identify objects, classify human motion, and so on. From his initial team of just a handful of Intel researchers, a user community grew to more than 47,000 people, and more than ten million copies of the toolset have been downloaded to date.

  Gary Bradski created a popular computer vision software library and helped design robots. He would later leave robotics to work with a company seeking to build augmented reality glasses. (Photo © 2015 by Gary Bradski)

  Realizing that he would one day leave Intel and would need a powerful toolset for his next project, Bradski developed a second agenda. OpenCV would be his calling card when he left his job at the chipmaker. Open-source software efforts were in favor inside Intel because the company wanted leverage in its difficult relationship with Microsoft. The two companies dominated the personal computing industry, but often clashed over issues of control, strategic direction, and ultimately revenue. For a while Bradksi had tremendous support inside the laboratory: at one point, he had fifteen researchers on the OpenCV project. That moment was one of the high points of his career at Intel.

  Then, Intel gave him a division award and told him, “All right, now you have to move on.” “What do you mean?” he responded to his managers. “This is a decadelong project.” Grudgingly, he did some other things, but he covertly kept the OpenCV project going on the side. That did not sit well inside the giant semiconductor firm. One of his Russian programmers was given a performance review demerit—“improvement required”—by management because he was associated with the program.

  Intel’s refusal to see the value of the project left Bradski feeling disaffected. In 2001, Intel dropped its camera division, which pushed him to the edge. “More shortsighted bean counter thinking,” he decided. “Of course this is low-margin silicon, but this is a loss leader, so you can eventually profit from the whole thing!” He had no idea that the mobile computing and smartphone wave was just a half decade away, but at that moment, he was right. Intel, in retrospect, had had a history of trying new ideas and then canceling them before they could bear fruit. His frustration made him an easy recruit for Sebastian Thrun, who was then building his team at Stanford to create the Stanley autonomous vehicle for the 2005 DARPA competition.

  They had struck up a relationship when Thrun had been at Stanford on sabbatical in 2001. When he returned in 2003 as a faculty member, Bradski, who was disaffected with Intel, was preparing to take his own sabbatical at EPFL, a Swiss research university in Lausanne. Thrun said, “Why don’t you come to Stanford instead?” Bradski was faced with a difficult decision. Switzerland would have offered him an academic feast, a chance to work on neural nets and evolutionary learning algorithms, and a great party. At the end of the day, he realized that a sabbatical at EPFL was a diversion for someone who had entrepreneurial aspirations, and the nightmare Swiss bureaucracy overwhelmed him: he should have started a year earlier getting his kids into private school, and renting a house in Lausanne was a challenge—one potential landlord told him there would be no showering after ten P.M. and he wouldn’t permit noisy children!

  So Bradski switched gears and took his sabbatical at relatively laid-back Stanford. He taught courses and flirted with ideas for a new start-up. His first project involved building an advanced security camera. However, he ended up with a partner who was a poor match for the project, and it quickly turned into a bad marriage. Bradski backed out. By that time, his sabbatical was over, so he went back to work at Intel and managed a large research group. He quickly realized that management involved a lot of headaches and little interesting work, so he tried to pare down his group to a core team.

  Before, Bradski had been oblivious to the frustrations of other researchers, but now he noticed that engineers everywhere inside the company had similar frustrations. He joined an underground laboratory for the disaffected. Then, on a visit to Stanford, Thrun said, “Come out back to the parking lot.” Thrun showed Bradski Stanley, the secret project preparing to enter the second DARPA Grand Challenge. This was obviously the coolest thing around, and Bradski immediately fell in love with the idea. Back at Intel, he quickly pulled together a secret skunkworks group to help with the computer vision system for the car. He didn’t bother to ask permission. He hosted his design meetings during lunchtime and met with the Stanford team on Tuesdays.

  There were immediately two problems. After Intel promised that it would not involve itself directly in the DARPA Grand Challenge, the company started sponsoring Red Whittaker’s CMU team. Bradski’s boss started getting complaints that Bradski was distracting people from their assigned work. “This could build up to be a firing offense,” his boss told him. “We’re not sponsoring the Stanford team and we’re not getting into robotics.” As a concession, Bradski’s boss told him he could continue to work on the project personally, but could not involve other Intel Labs researchers. By then, however, Bradski no longer cared about being fired. That made everything a lot easier, and the lunchtime meetings intensified.

  The tensions at Intel came to a head two days before the race. The cars a
nd teams had arrived in Primm, Nevada, a three-casino watering hole on the California-Nevada border. Bradski called a contact in Intel’s marketing department and said he needed an immediate decision about whether Intel was going to officially sponsor the Stanford car. A decal on the car would usually cost $100,000, but Thrun told him that Bradski’s team had donated so much volunteer labor that they could have the sponsorship for just $20,000. The Intel marketing guy loved the idea: sponsoring two cars would double Intel’s chance of backing a winner, but he balked at making an instant decision. “The money’s there, but I can’t just give it to you unilaterally,” the executive told him.

  “Look, the cars are about to be sequestered, we have half an hour left,” Bradski responded.

  It worked. “Okay, do it,” the executive said.

  Because it was so late, there was no room left on the car except a passenger window—a brilliantly visible location. Stanley won the race and Intel had backed a winner, so that was a coup. Bradski had pulled himself back from the edge of unemployment.

  The vision system contributed to Stanley’s success. The car relied on lasers that could sense a dynamic point cloud around the car and digital cameras that fed machine vision algorithms. In the end, the cameras saw far enough ahead that Stanley could maintain speed without slowing down. And going fast, needless to say, was necessary to win.

  The glory didn’t last long, however. Bradski had secured a small DARPA contract to research “cognitive architectures” with Thrun and Daphne Koller, another Stanford machine-learning expert. However, the DARPA program manager had announced his departure, which meant the grant was likely not be renewed, which in turn meant Bradski would have to look for funding elsewhere. Sure enough, Phase II was canceled as “too ambitious.”

  Bradski was very intrigued by robotics, so he used some of his grant money to purchase a robot arm. The $20,000 purchase set off a small explosion inside Intel’s legal department. The grant money, they insisted, was restricted for hiring interns, not buying hardware, and he had to transfer the ownership of the robot arm away from Intel. Bradski gave the arm to the Stanford STAIR project, which was run by Andrew Ng. Ng was starting to explore the world of robotics with machine-learning ideas. Could they design a robot to load and unload a dishwasher? It became part of the mix leading to the PR1 robot that was brewing between Salisbury’s laboratory and Ng’s project.

  Meanwhile, Bradski found Intel’s bureaucracy more and more overbearing. He knew it was time to leave and quickly negotiated a deal to join an Israeli machine vision start-up based in San Mateo. He took the OpenCV project with him. The machine vision start-up, however, turned out to be a less than perfect match. The Israelis loved conflict and Bradski was constantly butting heads with the CTO, a former sergeant in the Israeli Army. He would usually win the arguments, but at a cost. He began job hunting again after being at the new company for just a year.

  It was hard to search for a job clandestinely. He toyed with working at Facebook, who had offered him a job, but they weren’t doing anything interesting in computer vision. “Come anyway,” they told him. “We’ll find something for you to do.” To Bradksi, their recruiting seemed highly disorganized. He showed up for his interview and they told him he was late. He showed them the email that indicated that he was, in fact, on time.

  “Well,” they said, “you were supposed to be down the street an hour ago.”

  Down the street he found the building locked, closed, and dark. It occurred to him that perhaps this was some kind of weird job test, and that a camera might be following him to see what he was going to do. He kicked the door and finally someone came out. The man didn’t say anything, but it seemed obvious to Bradski that he had woken him up. The guy held the door open so Bradski could go inside, then walked off silently. Bradski sat down in the dark building and before long an admin arrived and apologized for being late. There was no record of a scheduled interview and so he called the recruiter who had supposedly set everything up. After a lot of apologizing and some more runaround, Bradski had his interview with Facebook’s CTO. A few days later, he had his second interview with a higher-ranking executive. The Facebook offer would have given him a lot of stock, but going to work for Facebook didn’t make much sense. Miserable with the Israelis, Bradski realized he would also be miserable at Facebook, where he would most likely be forced to work on uninteresting projects. So he kept hedging. The longer he held out, the more stock Facebook offered. At that point, the job was probably worth millions of dollars, but would cause Bradski great unhappiness in what seemed like a pressure cooker.

  One day, Andrew Ng called Bradski and told him he needed to meet an interesting new group of roboticists at a research lab called Willow Garage. Founded by Hassan, it was more of a research lab than a start-up. Hassan was preparing to hire seventy to eighty roboticists to throw things against the wall and see what stuck. It fit within a certain Silicon Valley tradition; labs like Xerox PARC and Willow Garage were not intended to directly create products. Rather they experimented with technologies that frequently led in unexpected directions. Xerox had created PARC in 1970 and Paul Allen had financed David Liddle to “do PARC right” when he established Interval Research in 1992. In each case the idea was to “live in the future” by building technologies that were not quite mature but soon would be. Now it looked like robotics was ripe for commercialization.

  Initially Bradski was hesitant about going by for a quick lunchtime handshake. He would have to race down and back or the Israelis would notice his absence. Ng insisted. Bradski realized that Andrew was usually right about these things and so decided to give it a shot. Everything clicked. At the end of the afternoon, Bradski was still there and he no longer cared about his start-up. This was where he should be. At the end of the day, while he was still sitting in the Willow Garage parking lot, he called Facebook to say he wasn’t interested. Shortly afterward, he quit his start-up.

  In December of 2007 Bradski was hired to run the vision group for the next generation of Salisbury and Ng’s earlier robot experiments, morphing PR1 into PR2. They built the robot and then ran it through a series of tests. They wanted the robot to do more than retrieve a beer from the fridge. They “ran” a marathon, maneuvering the robot for twenty-six miles inside the company office while Google cofounder Sergey Brin was in attendance. Afterward, they instructed the robot to find and plug itself into ten wall sockets within an hour. “Now they can escape and fend for themselves,” Bradski told friends via email.

  PR2 wasn’t the first mobile robot to plug itself in, however. That honor went to a mobile automaton called “The Beast,” designed at the Johns Hopkins Applied Physics Lab in 1960—but it could do little else.16 PR2 was Shakey reborn half a century later. This time, however, the robot was far more dexterous. Pieter Abbeel, a University of California at Berkeley roboticist, was given one of eight PR2s that were distributed to universities. With his students, he taught the machine to fold laundry—albeit very slowly.

  Though the Willow Garage team had made a great deal of progress, their research revealed to them just how far they were from developing a sophisticated machine that could function autonomously in an ordinary home. Kurt Konolige, a veteran SRI roboticist recruited by Bradski to Willow Garage, had told Bradski that these were decadelong technology development projects. They would need to refine each step dozens of times before they got everything right.

  In the end, however, like Paul Allen, who had decided to pull the plug on Interval Research after just eight years of its planned ten-year life span, Scott Hassan proved not to have infinite patience. Bradski and Konolige looked on in dismay as the Willow Garage team held endless brainstorming sessions to try to come up with home robot ideas that they could commercialize relatively quickly. They both realized the lab was going to be closed. Bradski believed he knew what people really wanted in their homes—a French maid—and that wasn’t going to be possible anytime soon. In his meetings with Hassan, Bradski pleaded for his team to be permi
tted to focus instead on manufacturing robotics, but he was shot down every time. Hassan was dead-set on the home. Eventually, Konolige didn’t even bother to show up at one of the meetings—he went kayaking instead.

  For a while Bradski tried to be a team player, but then he realized he was in danger of reentering the world of compromises that he had left at Intel.

  “What the hell,” he thought. “This isn’t me. I need to do what I want.”

  He started thinking about potential applications for industrial robotics integration, from moving boxes to picking up products with robot arms. After discussing robotics extensively with people in industry, he confirmed that companies were hungry for robots. He told Willow’s CEO that it was essential to have a plan B in case the home robot developments didn’t pan out. The executive grudgingly allowed Bradski to form a small group to work on industrial applications.

  Combining robot arms with new machine vision technology, Bradski’s group made rapid progress, but he tried to keep word of the advances from Hassan. He knew that if word got out, the project would quickly be commercialized. He did not want to be kicked out of the Willow Garage “nest” before he was ready to launch the new venture. Finally, early in 2012, one of the programmers blabbed to the Willow Garage founder about their success and the industrial interest in robotics. Hassan sent the group an email: “I will fund this tomorrow, let’s meet on Friday morning.”

  With Konolige and several others, and with start-up funding from Hassan, Bradski created Industrial Perception, Inc., a robotic arm company with a specific goal—loading and unloading boxes from trucks such as package delivery vehicles. After Bradski left to cofound Industrial Perception, Willow gradually disintegrated. Willow was divvied up into five companies, several robot standards efforts, and a consulting group. It had been a failure, but home robots—except for robotic vacuum cleaners—were still a distant goal.