In The Plex

by Steve Levy

  The India-born computer scientist had already been on Google’s radar: when he ate lunch at a joint called World Wraps in Palo Alto, he’d run into Sergey Brin, who would invariably hand him a business card and urge him to apply to Google. Bharat was impressed with Google—he’d actually presented his Hilltop algorithm in the same session at the conference in Australia when Brin and Page showed off Google to a bowled-over audience of IR people. He also liked Sergey. Their mutual friend Rajeev Mowani once hosted a seminar where Brin had arrived on Rollerblades and began rhapsodizing about PageRank without missing a beat. Bharat thought that was incredibly cool. But Google was so small. It was hard for Bharat to imagine leaving the creature comforts of a big company for an operation with a single-digit workforce located over a bicycle shop and decorated in a style that mixed high-tech Dumpster with nursery school. Plus he cherished the ability to pursue research, something he doubted was possible at a tiny start-up.

  Then Google hired Jeff Dean, and Bharat was stunned. It was like some basketball team playing in an obscure minor league grabbing a player who was first-round NBA material. Those guys were serious. Soon after, Bharat heard that this just-born start-up, which could barely respond to its query traffic, was starting a research group! It sounded improbable, but he climbed the flight of stairs in the Palo Alto office for an interview. Bharat said straight out that he was skeptical of Google’s research ambitions. From what he could see, there were a lot of people running around with pagers and flicking at their keyboards to keep the system going. “Larry, why do you say you want to do research?” he said to Page. “You are such a tiny group!” Page’s answer was surprising and impressive. Looking at things from a different perspective could lead to unexpected solutions, he said. Sometimes in engineering you look at things with tunnel vision and need a broader perspective. He told Bharat a story about Kodak that involved some seemingly intractable practical problem that was solved by an unexpected intervention from someone in the research division. Page wanted that kind of thing to happen at Google.

  That interaction sold Bharat. Here was a guy who was young, inexperienced, and probably half nuts—but technically adept and infectiously confident. “I could respect Larry in a way that I couldn’t respect people running other start-ups,” says Bharat. “I knew the technical content of his work.” What’s more, Bharat could feel the pull of Page’s crusade to make the world better by cracking hard problems at the intersection of computer science and metaphysics. Bharat had thought a lot about search and was enthralled with its mysteries. On the face of things, it seemed so tantalizingly easy. But people had grasped only the slightest fraction of what was possible. To make progress, even appreciate this space, you would have to live in the data, breathe them in like a fish passing water through its gills. Here was his invitation. Bharat would wind up working an evolution of his Hilltop algorithm, called web connectivity analysis, into Google’s search engine. It would be the company’s first patent.

  The same almost mystical attraction of Google’s ambitions led to another impressive hire in early 2000: Anurag Acharya, a Santa Barbara professor who was a colleague of Hölzle. Acharya, who’d gotten his PhD at Carnegie Mellon, had spent his entire life in academia but at age thirty-six had been questioning his existence there. He had tired of a routine where people took on a problem of limited scope, solved it, published the results, and then went on to the next. He remembered when he’d been a student and had sat with his adviser, a deep thinker who spent his entire life grappling with a single giant mystery: what is the nature of mind? More and more, Acharya thought that there was beauty in grappling with a classically hard problem that would survive after you leave the earth. Talking to Hölzle during an interview for this little company, he realized that search was that kind of problem. “I had no background in search but was looking for a problem of that kind,” he says. “It appeared that, yes, that could be it.” Adding to Google’s appeal was his own background—like several of his new colleagues, he was from provincial India. (And like many at Google, including the founders, his parents were academics.) He often thought of the people in his home country, who were not just poor but information-impoverished as well. “If you were successful at Google, people from everywhere would have the ability to find information,” he says. “I come from a place where those boundaries are very, very apparent. They are in your face. To be able to make a dent in that is a very attractive proposition.”

  Bharat recommended another friend named Ben Gomes, who worked at Sun. The two had studied for exams together as high school friends in Bangalore, India. Gomes joined Google the same week Bharat did. And Bharat had another friend who was among the best catches of all: Amit Singhal.

  Born in the Indian state of Uttar Pradesh, in the foothills of the Himalayas, Singhal had arrived in the United States in 1992 to pursue a master’s degree in computer science at the University of Minnesota. He’d become fascinated with the field then known as information retrieval and was desperate to study with its pioneering innovator, Gerard Salton. “I only applied to one grad school, and it was Cornell,” he says. “And I wrote in my statement of purpose that if I was ever going to get a PhD, it’s with Gerry Salton. Otherwise, I didn’t think a PhD was worth it.” He became Salton’s assistant, got his PhD at Cornell, and eventually wound up at AT&T Labs.

  In 1999, Singhal ran into Bharat at a conference in Berkeley. Bharat told him he was leaving DEC for an exciting start-up that wanted to take on the biggest problems in search. It had a funny name, Google. Singhal should work there, too. Singhal thought the idea was ridiculous. Maybe it was all right for Bharat, who was a couple of years younger and unmarried. But Singhal had a wife and daughter and a second child on the way. “These little companies are all going to die,” he said. “I work for AT&T—the big ship that always sails. I can’t go to Google-schmoogle because I have a family to support.”

  Not too long afterward, the big ship AT&T began to take on water. “In 2000, I was here,” says Singhal.

  In barely a year since Brin and Page had formed their company, they had gathered a group of top scientists totally committed to the vision of their young founders. These early employees would be part of team efforts that led to innovation after innovation that would broaden Google’s lead over its competitors and establish it as synonymous with search. But those breakthroughs were in the future. In 2000, those big brains were crammed into a single conference room working on an emergency infrastructure fix. Google had taken ill.

  The problem was the index storing the contents of the web in Google’s servers. For a couple of months in early 2000, it wasn’t updating at all. Millions of documents created during that period weren’t being collected. As far as the Google search engine was concerned, they didn’t exist.

  The problem was a built-in flaw in the crawling and indexing process. If one of the machines devoted to crawling broke down before the process was completed, indexing had to begin from scratch. It was like a role-playing computer game in which you would spend hundreds of hours building a character and then lose all that effort if your character got killed by a stray beast or a well-armed foe. The game world had learned to deal with the problem—dead avatars could be resurrected after a brief pause or an annoying dislocation. But Google hadn’t.

  The flaw hadn’t been so bad in the earlier days of Google, when only five or so machines were required to crawl and index the web. It was at least a ten-day process with one of Google’s first crawl engineers, Harry Cheung (everyone called him Spider-Man), at his machines, monitoring progress of spiders as they spread out through the net and then, after the crawl, breaking down the web pages for the index and calculating the page rank, using Sergey’s complicated system of variables with a mathematical process using something called eigenvectors, while everybody waited for the two processes to converge. (“Math professors love us because Google has made eigenvectors relevant to every matrix algebra student in America,” says Marissa Mayer.) Sometimes, because of quirks in
the way the web addresses were numbered, the system crawled the same pages and showed no movement, and then you’d have to figure out whether you were actually done or had hit a black hole. This problem, though, had been generally manageable.

  But as the web kept growing, Google added more machines—by the end of 1999, there were eighty machines involved in the crawl (out of a total of almost three thousand Google computers at that time)—and the likelihood that something would break increased dramatically. Especially since Google made a point of buying what its engineers referred to as “el cheapo” equipment. Instead of commercial units that carefully processed and checked information, Google would buy discounted consumer models without built-in processes to protect the integrity of data.

  As a stopgap measure, the engineers had implemented a scheme where the indexing data was stored on different hard drives. If a machine went bad, everyone’s pager would start buzzing, even if it was the middle of the night, and they’d barrel into the office immediately to stop the crawl, copy the data, and change the configuration files. “This happened every few days, and it basically stopped everything and was very painful,” says Sanjay Ghemawat, one of the DEC research wizards who had joined Google.

  “The whole thing needed rethinking,” says Jeff Dean.

  Actually, it needed redoing, since by 2000 the factors impeding the crawl were so onerous that after several attempts it looked as though Google would never build its next index. The web was growing at an amazing pace, with billions of more documents each year. The presence of a search engine like Google actually accelerated the pace, offering an incentive to people as they discovered that even the quirkiest piece of information could be accessed by the small number of people who would appreciate it. Google was trying to contain this tsunami with more machines—cheap ones, thus increasing the chance of a breakdown. The updates would work for a while, then fail. And now, weeks were passing before the indexes were updated.

  It’s hard to overestimate the seriousness of this problem. One of the key elements of good search was freshness—making sure that the indexes have recent results. Imagine if this problem had happened a year later, after the September 11, 2001, terrorist attacks. Doing a Google search for “World Trade Center” that November or December, you would have found no links to the event. Instead, you’d have results that suggested a fine-dining experience at Windows on the World, on the 107th floor of the now-nonexistent North Tower.

  A half-dozen engineers moved their computers into a conference room. Thus Google created its first war room. (By then—less than a year after moving from the house in Menlo Park to the downtown Palo Alto office—Google had moved once again, to a roomier office-park facility on Bayshore Road in nearby Mountain View. Employees dubbed it the Googleplex, a pun on the mathematical term googolplex, meaning an unthinkably large number.) When people came to work, they’d go to the war room instead of the office. And they’d stay late. Dean was in there with Craig Silverstein, Sanjay Ghemawat, and some others.

  They built a system that implemented “checkpointing,” a way for the index to hold its place if a calamity befell a server or hard disk. But the new system went further—it used a different way to handle a cluster of disks, more akin to the parallel-processing style of computing (where a computational task would be split among multiple computers or processers) than the “sharding” technique Google had used, which was to split up the web and assign regions of it to individual computers. (Those familiar with computer terms may know this technique as “partitioning,” but, as Dean says, “everyone at Google calls it sharding because it sounds cooler.” Among Google’s infrastructure wizards, it’s key jargon.)

  The experience led to an ambitious revamp of the way the entire Google infrastructure dealt with files. “I always had wanted to build a file system, and it was pretty clear that this was something we were going to have to do,” says Ghemawat, who led the team. Though there had previously been systems that handled information distributed over multiple files, Google’s could handle bigger data loads and was more nimble at running full speed in the face of disk crashes—which it had to be because, with Google’s philosophy of buying supercheap components, failure was the norm. “The main idea was that we wanted the file system to automate dealing with failures, and to do that, the file system would keep multiple copies and it would make new copies when some copy failed,” says Ghemawat.

  Another innovation that came a bit later was called the in-RAM system. This involved putting as much of the index as possible in actual computer memory as opposed to the pokier, less reliable hard disk drives. It sped things up considerably, allowed more flexibility, and saved money. “The in-memory index was, like, a factor of two or three cheaper, because it could just handle many, many more queries per machine per second,” says Dean.

  The system embodied Google’s approach to computer science. At one point, the cost of fixed memory (in chips as opposed to spinning hard disks) would have been so expensive that using it to store the Internet would have been a daffy concept. But Google’s engineers knew that the pace of technology would drive prices down, and they designed accordingly. Likewise, Google—as its very name implies—is geared to handling the historic expansion of data that the digital revolution has triggered. Competitors, especially those who were successful in a previous age, were slow to wrap their minds around this phenomenon, while Google considered it as common as air. “The unit of thinking around here is a terabyte,” said Google engineering head Wayne Rosing in 2003. (A terabyte is equal to around 10 trillion bits of data.) A thirty-year Silicon Valley veteran whose résumé boasted important posts at DEC, Apple, and Sun, Rosing had joined Google in 2001 in part because he saw that it had the potential to realize the vision of Vannevar Bush’s famous memex paper, which he had read in high school. “It doesn’t even get interesting until there’s more than many terabytes involved in problems. So that drives you into thinking of hundreds of thousands of computers as the generic way to solve problems.” When you have that much power to solve problems, you have the ability to do much more than solve them faster. You can tackle problems that haven’t even been considered. You can build your own paradigms.

  Implementing the Google File System was a step toward that new paradigm. It was also a timely development, because the demands on Google’s system were about to increase dramatically. Google had struck a deal to handle all the search traffic of Yahoo, one of the biggest portals on the web.

  The deal—announced on June 26, 2000—was a frustrating development to the head of Yahoo’s search team, Udi Manber. He had been arguing that Yahoo should develop its own search product (at the time, it was licensing technology from Inktomi), but his bosses weren’t interested. Yahoo’s executives, led by a VC-approved CEO named Timothy Koogle (described in a BusinessWeek cover story as “The Grown-up Voice of Reason at Yahoo”), instead were devoting their attention to branding—marketing gimmicks such as putting the purple corporate logo on the Zamboni machine that swept the ice between periods of San Jose Sharks hockey games. “I had six people working on my search team,” Manber said. “I couldn’t get the seventh. This was a company that had thousands of people. I could not get the seventh.” Since Yahoo wasn’t going to develop its own search, Manber had the task of finding the best one to license.

  After testing Google and visiting Larry Page several times, Manber recommended that Yahoo use its technology. One concession that Yahoo gave Google turned out to be fateful: on the results page for a Yahoo search, the user would see a message noting that Google was powering the search. The page even had the Google logo. Thus Yahoo’s millions of users discovered a search destination that would become part of their lives.

  As part of the deal, Google agreed to update its index on a monthly basis, something possible after the experience in the war room. Google now had the most current data in the industry. It also boasted the biggest index; on the day it announced the Yahoo deal, Google reported that its servers now held more than a billi
on web pages. This system remained state of the art until the summer of 2003, when Google launched a revamp of its entire indexing system to enable it to refresh the index from day to day, crawling popular sites more often. The code name for the 2003 update was BART. The title implied that Google’s system would match the aspirations (if not the accomplishments) of the local mass transit system: “always on time, always fast, always on schedule.” But the code name’s actual origin was an engineer named Bart.

  Even though Google never announced when it refreshed its index, there would invariably be a slight rise in queries around the world soon after the change was implemented. It was as if the global subconscious realized that there were fresher results available.

  The response of Yahoo’s users to the Google technology, though, was probably more conscious. They noticed that search was better and used it more. “It increased traffic by, like, 50 percent in two months,” Manber recalls of the switch to Google. But the only comment he got from Yahoo executives was complaints that people were searching too much and they would have to pay higher fees to Google.

  But the money Google received for providing search was not the biggest benefit. Even more valuable was that it now had access to many more users and much more data. It would be data that took Google search to the next level. The search behavior of users, captured and encapsulated in the logs that could be analyzed and mined, would make Google the ultimate learning machine.