How the Internet Happened

by Brian McCullough

  Far from being a fad, the habits Americans acquired during the bubble era ingrained themselves into the rhythms of everyday life. The dot-coms, the training wheels for the Internet, the pioneers, they all taught us to live online. We all might have jumped from dial-up to broadband, but few of us quit using the net. There was no going back.

  And even as the dot-com companies were crashing and burning, there were already new innovators on the scene who would move the Internet forward in an entirely new, entirely personal, and (finally) exceedingly profitable way.

  11

  I’M FEELING LUCKY

  Google, Napster and the Rebirth

  When Larry and Sergey first met, they didn’t like each other much.

  In the summer of 1995, Larry Page was considering a transfer to Stanford University’s graduate program in computer science. Sergey Brin was already two years into the program, and he had signed up to be a tour guide of sorts to potential students. One summer day, he showed Page and a group of other potential Stanford students around the Bay Area.

  “I thought he was pretty obnoxious,” Page said later of his guide. “He had really strong opinions about things and I guess I did, too.”

  “We both found each other obnoxious,” Brin agrees. They might have stepped on each other’s toes a bit, but at the same time there was a degree of frisson to the encounter. “We spent a lot of time talking to each other,” Brin would recall, “so there was something there. We had a kind of bantering thing going.”1

  On the surface, it might not have seemed like Page and Brin had anything in common. Page was a midwesterner, born in East Lansing, Michigan, on March 26, 1973. Brin was born in Moscow, in the Iron Curtain–era USSR, on August 21, 1973, and was brought to the United States when he was six years old. Page was reserved, quiet, contemplative. Brin was outgoing, gregarious, loud. Page was a deep thinker, a visionary. Brin, a problem solver, an engineer’s engineer.

  But the two had more in common than anyone knew that first day. They both came from academic families. Page’s father was a pioneering computer science professor at Michigan State University, where his mother was also a computer programming instructor. Brin’s father was a mathematics professor at the University of Maryland and his mother a researcher at NASA’s Goddard Space Flight Center. Larry and Sergey both grew up to respect research, academic study, mathematics and especially computers. And they both had inquisitive minds, believing in the power of knowledge to overcome any obstacle, intellectual or practical. Each had been inculcated into this spirit of intellectual fearlessness at a young age.

  “You can’t understand Google,” early Google employee Marissa Mayer (and later, Yahoo CEO) has insisted, “unless you know that both Larry and Sergey were Montessori kids. It’s really ingrained in their personalities. To ask their own questions, do their own things. Do something because it makes sense, not because some authority figure told you. In a Montessori school, you go paint because you have something to express or you just want to do it that afternoon, not because the teacher said so. This is baked into how Larry and Sergey approach problems. They’re always asking, why should it be like that? It’s the way their brains were programmed early on.”2

  For Larry and Sergey, their intellectual fearlessness overlapped in such a way that their conflicting personalities actually ended up complementing each other. When Page came to Stanford for the 1995–96 academic year, he and Brin became close. Friends took to calling the duo LarryandSergey, and the pair would end up debating endlessly on topics ranging from philosophy to computing to films, two equally matched polymaths thrilling to the intellectual joust. Brin’s hobby project was creating a software program that could provide movie recommendations based on the tastes and viewing habits of other people who had seen similar films (not unlike what Netflix later perfected). Page’s dream obsession was creating a system of networked, autonomous cars to ferry people around.

  Even though they were the same age, Brin was academically two years ahead of Page because he had completed his undergraduate computer science degree at age nineteen and aced all of Stanford’s required doctoral program exams on the first try.3 But despite this head start, and despite being the recipient of a National Science Foundation fellowship that allowed him to do basically anything he wanted, Brin had stalled out in his quest to nail down a dissertation topic. Of course, the newly arrived Page also needed to decide on his dissertation, and so fate pushed the pair even closer together. In January 1996, LarryandSergey ended up working in the same office, number 360, in the just-completed William Gates Computer Science Building on Stanford’s campus. The building was of course named after the founder of Microsoft, who had donated $6 million to the construction. All his career, Bill Gates repeatedly predicted that one day, some student somewhere would found a company that would challenge Microsoft for dominance of the tech industry. His prediction turned out to be right, and from a building with his name on it.

  ■

  PAGE WAS STRUCK by a fundamental truth about the web that is glaringly obvious when you state it out loud: it is built on links. One page linking to another; one idea linking to another. As of yet, no one had bothered to analyze the structure of the link ecosystem in a comprehensive way. For example, it was possible to know that webpage A linked to webpage B because you could see it—you could follow the link. But what about the reverse? What pages had linked webpage A? There was no way to know. You couldn’t follow a link stream backward, only forward. Page wondered: if you analyzed all of the back links, if you mapped out the link structure of the entire web, what sort of insight might that data give you?

  Page’s intuition was that this might be more than just an interesting theoretical question. As he mulled over the idea with Brin, their shared upbringing as the children of academics kicked in. LarryandSergey knew the power of the academic citation. Their parents had published academic papers. They, themselves, intended to publish academic papers in order to earn their Ph.D.’s. And they knew that any academic paper worth its salt built its argument by citing other academic papers and studies. In the world of academia, those citations, the accumulated number of “votes” from paper to paper, served, over the years, to accrue value to given ideas—to essentially rank them based on the number of citations. The most cited papers were understood to be the most authoritative. “It turns out, people who win the Nobel Prize have citations from 10,000 different papers,” Page would say later.4

  Well, what was a web link but a digital citation? If you analyzed the links, analyzed the citations, you might be able to make inferences about the relative value of a given web page, and possibly even determine which webpage was more authoritative by analyzing the back links in the same way that counting the citations told you which academic paper was the definitive one. Larry Page wanted to map out the value of the web’s connections by going backward through the link chain. Page went to his academic advisor, Terry Winograd, and asked for the money and machines that would allow him to map the web’s links. He dubbed the project BackRub. When asked how much of the web he intended to map, he replied: “the whole web.”5

  So, in March of 1996, Larry Page launched BackRub by sending search bots, known as “spiders,” out into the web to find all the links. He started with a single page—the Stanford computer science department homepage—and then fanned out, following link after link, cataloging them all, and then ranking web pages based on these link citations. It was the mathematical complexity of this ranking—the complicated problem of determining which page was more valuable based on a combination of accumulated links as well as the authority passed through from pages that linked to other pages—that drew Sergey Brin to join the project. Larry and Sergey called their combined citation-ranking system PageRank, either as an ode to Page himself or as an obvious descriptor of what the system was intended to do.

  “The idea behind PageRank was that you can estimate the importance of a web page by the web pages that link to it,” Brin says. “We actually developed a
lot of math to solve that problem. Important pages tended to link to important pages. We convert the entire web into a big equation with several hundred million variables which are the PageRanks of all the web pages, and billions of terms, which are all the links.”

  “It’s all recursive,” Page said. “In a way, how good you are is determined by who links to you and who you link to determines how good you are. It’s all a big circle.”6

  LarryandSergey suddenly had a project that would make for a pretty interesting dissertation. And as soon as the pair looked at their results, they realized their intuition was dead on: the citation analogy worked. If you wanted to find the most authoritative webpage about a topic such as, say, windsurfing, BackRub/PageRank could tell you. It would know based on the accumulated links, of course, but also from the authority passed on from other authoritative sites. Thanks to Brin’s math (largely linear algebra and something about the eigenvector of a weighted link matrix, for those who know what that means), citations from obviously important websites were more valuable than others. A link from some unknown person’s personal webpage might be valuable, but a link from a professional windsurfer would be judged to be even more valuable— and a link from, say, Yahoo’s homepage would be even more valuable still.

  It was at this point that the really interesting application for this little math project became obvious. “It was pretty clear to me and the rest of the group,” Page said later, “that if you have a way of ranking things based not just on the page itself but based on what the world thought of that page, that would be a really valuable thing for search.”

  ■

  IT TURNED OUT THAT the reason search engines had never worked very well prior to PageRank was not that they were broken, but because they were missing the key innovation that Brin and Page had stumbled upon: relevancy. If, in 1997, you did a search for “automobile company” on even the best search engine at the time (AltaVista) you’d find yourself disappointed because the websites of Ford, General Motors or Toyota would probably not show up. It’s not that AltaVista couldn’t find those sites. It most certainly had! Ford.com or GM.com or Toyota.com were most likely in the list of tens of thousands of results that AltaVista had found. It was just that AltaVista had no way of surfacing those most relevant results to the top. So they were on page 3 of the search results. Or page 300.

  PageRank solved this problem. PageRank knew which sites were the most authoritative automotive sites already, and so when you combined its algorithmic prowess with the traditional tricks of information retrieval that all the search engines were already using, suddenly it all just worked. Indeed, as Page and Brin combined BackRub and PageRank with traditional search methods like analyzing on-page text, webpage titles or metatags and, especially, parsing the so-called anchor text of a link (someone who makes a link out of the words “flower shop” and then points it to a given website is really trying to tell you something), they realized PageRank was incredibly powerful. Page and Brin discovered that their algorithm was indeed recursive, meaning that the more data they fed it, the more webpages it analyzed, the better it got. By tweaking the math even more, LarryandSergey’s search tool could reliably find people, locate the most obscure fact or data, and even answer questions. PageRank wasn’t finding new things. It was merely finding things in a better way. The earlier search engines were already answering every query correctly. But it was finding the needle in the haystack and putting it at the top of the list that PageRank did better.

  “It wasn’t that they [Page and Brin] sat down and said, ‘Let’s build the next great search engine,’ ” said Rajeev Motwani, who was Brin’s academic advisor. “They were trying to solve interesting problems and stumbled upon some neat ideas.”7

  ■

  IT WAS A GOOD THING Page and Brin had not set out to build the next great search engine, because, at the time, no one was really clamoring for one. In the late nineties, when Page and Brin began refashioning BackRub/PageRank into a search engine, there was a universe of major search players: Yahoo, Excite, Lycos, AltaVista, AskJeeves, MSN, and on and on. In a time when Yahoo had a $100 billion market cap, who needed another entrant into an already-crowded space, no matter how superior it was? Fortunately, Page and Brin were not business-focused at that time. They were academics, more interested in defending a dissertation and publishing a paper on their research than starting a company around their idea.

  So, they produced that paper: “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” which was presented at a conference in Australia in May of 1998. But if Page and Brin initially stayed true to their chosen academic paths, that did not mean they were blind to the financial possibilities inherent in their work. How could they have been? Students studying computer science in the heart of Silicon Valley couldn’t help but notice what was going on all around them. “It was a hard time to stay in grad school,” remembered Tamara Munzner, one of the students sharing room 360 of the Gates Building with Page and Brin. “Every time you went to a party, you had multiple job offers and they were all real. I had to redecide every term not to leave.”8

  The obvious move was to license PageRank to one of the existing players, and indeed, this is what Page and Brin attempted to do. They met with everyone from the Yahoo founders Jerry Yang and David Filo, to another search pioneer, Infoseek’s Steve Kirsch. No one was interested. The closest they came to making a deal was when Page wrote up an extensive proposal to Excite’s leadership, suggesting they replace Excite’s existing algorithms with his. Doing so, he calculated, would generate an additional $47 million in revenue for the search engine. “With my help,” Page wrote in his proposal, “this technology will give Excite a substantial advantage and will propel it to a market leadership position.”9 All he asked for in exchange was a seemingly reasonable $1.6 million in cash and Excite stock—a nice little payday—and then he and Brin would return to finishing their doctorate work. Excite countered with $750,000, which Page and Brin rejected.

  The incumbent search players’ failure to scoop up the Page­Rank technology has become infamous in business lore as one of the great missed opportunities of all time. Larry Page has, on a few occasions, suggested that the search companies were simply myopic. “They were becoming portals. We probably would have licensed it if someone gave us the money. . . . [But] they were not interested in search,” Page has said. “They did have horoscopes, though.”10 But Excite CEO George Bell has a slightly different recollection: “The thing that Larry insisted on, that we all do recall, is that Larry said, ‘If we come to work for Excite, you need to rip out all the Excite technology and replace it with [our] search.’ And, ultimately, that’s—in my recollection—where the deal fell apart.”11 This was Page and Brin’s intellectual fearlessness demonstrating itself for the first time in a competitive setting. The pair believed—knew—that they had a superior way of doing things, and so they thought nothing of going to an established search company and telling them their existing product sucked. This brashness had the effect of insulting Excite. Excite was a company founded by brilliant Stanford computer scientists, after all. “We had hundreds of engineers at that point,” Bell points out. Why should the company furlough their engineers just because two other engineers had come along with claims to be more brilliant? Bell claims that there was no way he could justify upsetting his existing talent, especially when some of them were founders of the company. “Ultimately I couldn’t stomach the cultural risk that Larry insisted on,” Bell says.

  But if Page and Brin were confident almost to the point of being arrogant, they certainly had plenty of data to back them up. In order to fine-tune their algorithm, the pair had needed plenty of real-world feedback. Starting in 1997, they had made the search engine available, first on Stanford’s internal network, and then to the general public. Through nothing but word of mouth, the service grew increasingly popular, serving more than 10,000 queries a day by late 1998.12 Page and Brin monitored the server logs and made tweaks to their system ba
sed on the data this provided. They named the service Google, a play on the word “googol,” which is a 1 followed by 100 zeros. The idea was to suggest they were capturing the whole web, everything in existence. “The name reflected the scale of what we were doing,” Brin said later.13 Googol.com was not available, so Google.com became the URL of the public service.

  The popularity of the service, combined with the vast computing resources eaten up by the spidering and indexing, meant that the Google project was rapidly outgrowing the scope of a simple research project. Even when it was housed on a single machine in a Stanford dorm room, Google was hogging large amounts of the university’s bandwidth. Stanford was, as ever, incredibly accommodating to an idea born within its walls, but the institution’s generosity had a practical and obvious ceiling.

  It was clear that if they wanted the Google experiment to continue, Page and Brin would need more resources. More computers, more bandwidth, more people to work on the algorithm—this all meant more money than a research budget, even a generous one, could provide. So the pair turned to another Stanford faculty advisor, David Cheriton. Cheriton introduced the pair to Andy Bechtolsheim, a successful entrepreneur who had founded Sun Microsystems while also a Ph.D. student at Stanford. One morning in late 1998, Page and Brin met Bechtolsheim at Cheriton’s home. Bechtolsheim made out a check on the spot for $100,000 in the name of Google Inc. The check sat in Page’s dorm room desk for a number of weeks before Google Inc. was formally incorporated on September 7, 1998. Page and Brin would raise an additional $1 million when David Cheriton kicked in some money, as well as a few others, including former Net­scape executive Ram Shriram and Jeff Bezos of Amazon.

  Page and Brin were now entrepreneurs, if perhaps still a little reluctantly. But they were not entrepreneurs in the mold of so many others in the dot-com era. Rather than blowing Google’s funds on lavish launch parties or marketing campaigns, they stayed grad students at heart, and instead invested all the money they had raised in continuing their project efficiently. Instead of building out their system by buying software from Microsoft, they used the free Linux operating system. Instead of splurging $800,000 on setups from IBM or Oracle, they spent a mere $250,000 to cobble together a rack of eighty-eight computers to meet their number-crunching needs. At Stanford they had begged, borrowed, and almost quite literally stolen the computers they needed to keep Google running. Now, they simply switched to buying computers off the shelf from Fry’s, the famous Silicon Valley electronics store, and fashioned them into a strung-together system of their own design. Part of this was simple frugality, a habit that would serve them well when the dot-com bubble burst a few short years later. But a lot of it was Page and Brin’s ingrained Montessori philosophy: they never met a problem they couldn’t solve through smart engineering.