Valley of Genius

by Adam Fisher

  Scott Hassan: So what Larry did was he started writing a web crawler. So what a web crawler does is that you give it a starting page and then it downloads that page, looks through the page, finds all the hyperlinks, then it downloads them and then it just continues to do that. And so that’s how a web crawler works.

  Terry Winograd: Downloading the entire web was their goal. Of course in those days “the entire web” meant something very different. And there was a technical question of how much space it would take? How many machines? How much bandwidth? I mean, one of the things we were fighting is the amount of just plain network bandwidth, out of Stanford, that it took to do all that crawling.

  Scott Hassan: So if you were to write a crawler in a very simple form, it will be super-super-slow, because you’ll be fetching a page and you’ll be waiting for that thing to come back, right? At the time there were millions of web pages already, and so if it took you a minute to download every page, there would be more pages being created per minute than you would download per minute, so you would never finish.

  Terry Winograd: Getting hundreds of thousands of pages and downloading them was a big deal.

  Scott Hassan: So in the fall of ’95 for some reason I started hanging out with Larry in his office. It was a really big office down near Winograd’s, and he was in there working on it. And at that time I got really careful helping people with their projects because I found that once I started helping them, I started becoming their programmer. So I didn’t want to help Larry specifically with his program—but he was having problems. He had written his crawler in a language called Java. Now, Java had just come out of SUN and was extremely buggy, especially running on top of Linux. So I carefully just helped Larry fix the Java runtime environment and the Linux kernel—rather than helping Larry with his actual Java code.

  Terry Winograd: There’s a pecking order in graduate school. So, PhD students are expected to do “original research” and master’s students are not. They’re expected to take courses and learn what to do and be helpers on programs. Scott was a master’s student.

  Scott Hassan: At the time, Larry was trying to download a hundred pages simultaneously. And I was fixing some of the bugs that he was having with Java itself, and this went on for weeks, if not months. And I remember thinking, Wow, this is insane! because I was spending a lot of time fixing this underlying tool. And so one weekend, I just took all his code, I took his whole entire thing, and threw it all out, and rewrote the thing that he’s been working on for months very quickly—over a weekend—because I was just sick and tired of it. I knew I could get the thing working if I used a language I knew very well called Python. I wrote it in such a way that it could download thirty-two thousand pages simultaneously. So Larry went from barely downloading a hundred, to doing thirty-two thousand simultaneously on a single machine.

  Terry Winograd: Scott was a programmer. I was not in the meeting so I don’t know, but the basic model is Larry said, “Okay, we need a piece of code that does X, Y, Z,” and Scott went off and built it.

  Scott Hassan: I was really happy to show it to him on Monday, but Larry took one look at it and goes, “Great, it looks like you have this problem here, you have this problem here…” He pointed out like three different problems immediately. So very quickly it turned into him telling me what’s wrong with it, and then me fixing it: the thing that I was trying to avoid in the first place.

  By March 1996, Hassan’s new and improved crawler had churned through fifteen million web pages: not the entire internet, but a sizable chunk. It was now possible to surf the web backward. The question then becomes: Why? What could back links tell you? What was this information good for?

  Larry Page: Amazingly, I had not thought of building a search engine. The idea wasn’t even on the radar.

  Scott Hassan: So at the time he started doing this, there were already a few search engines out there.

  John Markoff: The Yahoo model was actually to try to structure the information on the internet by sorting it—by hand—into categories. Then people created search interfaces that would allow you to type an arbitrary string and get back a list of results.

  Brad Templeton: Then Alta Vista launched in December of 1995 and it immediately became the popular favorite. Everybody switched very quickly, because of its speed.

  Scott Hassan: And so nobody wanted to do a search engine, because that wasn’t “research.” At Stanford you had to do something fundamentally new, so you couldn’t do something that’s already done, because that’s not cool, right?

  John Markoff: There were so many search engines at the time. They were all over the place. Building the crawler and downloading the web was not Google’s breakthrough. The breakthrough was PageRank.

  Terry Winograd: I can remember Larry talking about a random walk on the web. “Random surf,” he called it. So, you’re on the web at some page and it’s got a bunch of links. So you pick one of them at random and you go there. And then you do this again and again with a zillion bots. So, if everybody’s doing this, where would you end up most of the time? The point is if lots of people point to me, you’re going to end up with me more often. I’m very important—so I’m getting a lot of traffic. Then if I point to you, you’re going to get a lot of them even if there’s only one link from me to you: You’re going to get a lot because I’m getting a lot. So think of this traffic moving through this network, just statistically. Who would get the most traffic?

  Scott Hassan: Larry came up with the idea of doing random walk, but Larry didn’t know how to compute it. Sergey looked at it and said, “Oh, that looks like computing the eigenvector of a matrix!”

  Sergey Brin: Basically we convert the entire web into a big equation, with several hundred million variables, which are the page ranks of all the web pages and billions of terms, which are the links. And we’re able to solve that equation.

  Terry Winograd: You can get fancy about it in formal terms. But in informal terms, PageRank was the implementation of that intuition.

  Scott Hassan: So Sergey just saw that and was like, “Okay great, I’m going to need a computer with four gigabytes of main memory to compute this.” So at the time to have a computer with four gigabytes of main memory was crazy, but it turns out that there was one computer in the computer science department that did have that, and that was in the graphics lab. They had a really, really big machine that had a whole bunch of memory and they were using it for graphics. So Larry got access to that big computer and we basically ran the algorithm on it for a couple of hours, and once it computed it, it was done.

  Sergey Brin: Every web page has a number.

  Larry Page: Then we were like, “Wow, this is really good. It ranks things in the order you expect them!”

  John Markoff: It’s a very simple idea: You saw the most popular things first. PageRank was an algorithm that looked at what other humans thought was significant—as demonstrated by other people linking to them—and used that as a mechanism for ordering search results.

  Sergey Brin: And we produced a search engine called BackRub. It was fairly primitive, it only actually looked at the titles of the web pages, but it was already working better than the available search engines at the time in terms of producing relevant results. If you’re searching for Stanford you get the Stanford home page back, for example.

  Scott Hassan: So then I sat everyone down and said, “Hey, let’s build a full search engine!” Right? And both Larry and Sergey thought it was going to be a lot of work. I was like, “No, no, no, actually it’s not that much work. I know exactly how to do it.”

  Butler Lampson: A search engine has two pieces. One piece is the piece that crawls the web and collects all the pages and the other is the piece that indexes them.

  Scott Hassan: If you look at the back of a book, sometimes there’s an index, and essentially that’s what you do when you build a search engine, is you do that, but you do that for every word.

  Butler Lampson: And then of course nowadays th
ere is a third piece which does the relevance part. It has to figure out what answers to show in response to the query.

  Scott Hassan: Very quickly, in six to eight weeks, we were able to build the whole structure of Google. It was mostly just Sergey and I from two a.m. to six a.m. in the morning. We just worked on it in the middle of the night, mainly because if I worked on it during the day I would get yelled at by my boss, because building a search engine was not considered research.

  Sergey Brin: By 1997 we are already developing a bunch of other technologies, too. For example, hypertext analysis, which is where we started asking questions like: Where are all the terms on the page? What are their font sizes? Which things are headers? And even: What is the text of pages which are closely linked to this page? What do they say? PageRank was the importance component, but we combined that with this relevance measure, based on this hypertext analysis, and the two together really functioned well.

  Scott Hassan: We got the search engine to a certain point and then Larry built this little interface. You go to this web page, and then on top of the web page there was a single box, very similar to Google’s search box today, right? It’s just a single box, and beside the box was another drop-down box, and “Which search engine do you want to use?”

  Brad Templeton: There was a bunch: Excite, Lycos, AltaVista, Infoseek, and Inktomi—that one was done at UC Berkeley.

  Scott Hassan: So, you could select one of those other search engines, and then you’d type in your little query and you’d hit Search, and what it would do, it would split the screen in half. On the left-hand side it would just pose that query to the search engine you chose, and then on the right-hand side it would then pose it to our search engine, so that you could compare the results side by side. So Larry set up all these meetings with all the search engine companies to try to license PageRank to them.

  David Cheriton: I think it must have been the ’97 time frame that Larry and Sergey came to me about the idea of licensing the search technology, and I told them that I didn’t think that this was likely to work, because the reality is it’s often difficult to find anybody in a company who is going to say, “Gee, this is really important and I’m willing to say we’re too dumb to have developed this ourselves, so we’ll license it from you!”

  Scott Hassan: Larry wanted to finish his PhD, so he just wanted to license it to whoever wanted it.

  David Cheriton: So anyway, they went off on that particular venture of trying to license it, and I think some of the interesting early stories of Google date from then—who could have bought the whole thing for $2 million or something of that flavor—if they had decided to pounce on it.

  Scott Hassan: I remember going to this one meeting at Excite, with George Bell, the CEO. He selects Excite and he types “internet,” and then it pops up a page on the Excite side, and pretty much all of the results are in Chinese, and then on the Google side it basically had stuff all about NSCA Mosaic and a bunch of other pretty reasonable things. George Bell, he’s really upset about this, and it was funny, because he got very defensive. He was like, “We don’t want your search engine. We don’t want to make it easy for people to find stuff, because we want people to stay on our site.” It’s crazy, of course, but back then that was definitely the idea: Keep people on your site, don’t let them leave. And I remember driving away afterward, and Larry and I were talking: “Users come to your website? To search? And you don’t want to be the best damn search engine there is? That’s insane! That’s a dead company, right?”

  Sergey Brin: Ironically, toward the end of the 1990s most of the portals started as search engines. Yahoo was the exception, but Excite, Infoseek, HotBot, and Lycos began as search engines.

  David Cheriton: Search kind of got steered off the path once the businesspeople got involved. They decided that the model that they understood was a newspaper. And so in a newspaper, search is a minor thing, it’s not the business. It’s the index that says where the sports section is and so on.

  Sergey Brin: Searching was viewed as just another service, one of a hundred different services. With a hundred services, they assumed they would be a hundred times as successful.

  David Cheriton: So I think it was approximately a year later that they came back to me and said they weren’t getting anyplace with the licensing, and I didn’t say, “I told you so,” but I think I felt a little bit smug inside.

  Terry Winograd: By that time, I think it was clear that they had traction in the real world and they wanted to put their energy into making that grow, not into doing an experiment that would satisfy some thesis committee.

  Sergey Brin: It was the summer of 1998. At that point we were just scrounging around to find resources, we had stolen these computers from all over the department, sort of. We’d assembled them all together, but they were haphazard, like a SUN, an IBM A/X computer, a couple PCs.

  Heather Cairns: They were taking servers off the loading dock. They were crashing them with traffic just through word of mouth.

  Larry Page: We caused the whole Stanford network to go down. For some significant amount of time nobody could log in to any computers at Stanford.

  Heather Cairns: And they were, essentially, nicely asked to leave because of that.

  Larry Page: Stanford said, “You guys can come back and finish your PhDs if you don’t succeed.”

  David Cheriton: They thought they had a big challenge of raising money, I thought that money wasn’t the big problem, and so I kind of put myself in the spot of having to prove this, so I contacted Andy Bechtolsheim.

  Sergey Brin: One of the founders of SUN Microsystems, and a Stanford alum.

  David Cheriton: Andy said he was interested, and we arranged to meet at my house, and Andy pulled up in his Porsche and they put on this demonstration of the Google search engine, and there was a certain amount of discussion.

  Larry Page: It seemed like they really wanted us to start a company.

  Andy Bechtolsheim: The question, of course, is, “How do you make money?” And the idea is, “Well, we’ll have these sponsored links and when you click on a link we’ll collect five cents.” And so I made this quick calculation in the back of my head: Okay, they are going to get a million clicks a day at five cents, that’s fifty thousand dollars a day—well, at least they won’t go broke.

  David Cheriton: Andy just got up and walked back to his Porsche, if not ran, from the porch to the Porsche, that is, got the checkbook, came back, and wrote them this check.

  Sergey Brin: He gave us a check for a hundred thousand dollars, which was pretty dramatic. The check was made out to “Google Inc.,” which didn’t exist at the time, which was a big problem.

  Larry Page: We didn’t have a checking account, we didn’t have a company, we didn’t have anything.

  David Cheriton: Andy just handed them the money: “Let’s work out the details later; let’s get going.”

  Sergey Brin: We hadn’t really discussed valuations and stuff like that. He figured it would pay off, and he was right. We finalized all the details on the round after that. I guess we figured if we didn’t agree later, that it would be a loan. He liked us and he just wanted to sort of push us forward.

  Larry Page: It was pretty unreal.

  Sergey Brin: It was like, That was pretty easy!

  Brad Templeton: Then they went to Burning Man.

  Ray Sidney: Sergey put up a Burning Man logo on the website. It was the first Google Doodle.

  Marissa Mayer: It was more of an out-of-office notification than anything else—it said, “We’re all at Burning Man.”

  Brad Templeton: There was a Google contingent that camped at Burning Man. I remember saying something rude to Marissa that I shouldn’t have said about wanting to see her naked. She won’t remember that, I hope.

  Marissa Mayer: Remember, we were all young, and we were all, in addition to being coworkers, friends.

  Scott Hassan: I was responsible for the shelter and Sergey was responsible for the food. So he we
nt to the Army Navy supply store and just bought all the rations—MREs. They were pretty fascinating. You would pour water in this little bag, and it had some sort of chemical in there, and it got really, really hot—and cooked it. So you didn’t even have to have a stove, you didn’t have to have anything! And we drove his car to Burning Man and just went around.

  Jamis MacNiven: In ’98 it was only about fifteen thousand people and really relaxed and cool and there was a lot of space—very spread out.

  Scott Hassan: We didn’t really have a camp, we had little tents and saw all the sites and stuff like that.

  Brad Templeton: In the early years Sergey would just actually sleep in whatever camp he found himself in at the end of the night.

  Google is incorporated immediately after the return from Burning Man.

  Heather Cairns: They handed me a folder full of checks for like $100,000, $200,000 from Andy Bechtolsheim, Jeff Bezos, David Cheriton. They sat in the back of my car for weeks because I couldn’t get out of work in time to even get a bank account opened.

  Ray Sidney: I had never worked at an early stage start-up before, and you know what? It was intense. My first week at Google I did two all-nighters. We saw this big opportunity and at the same time there was so much in doubt, so we wanted to do whatever we could to make it work, and so we worked hard. We had visions of greatness.

  Kevin Kelly: I had a chance to meet Larry Page because his brother, Carl, worked in the same building as Wired, and so I was at a party with Larry, and this is at a time when Google was pre-IPO and they actually didn’t have a business model, they weren’t selling ads…

  Heather Cairns: We didn’t have a business plan, and they would tell me that their actual mission statement was “to rule the Earth.” I’m thinking, Well, whatever you want, just make sure to sign my check and I’ll go on my way when it crashes and burns in a couple of years.

  Kevin Kelly: I said, “Larry, I don’t get it. What’s the future of search for free? I don’t see where you’re going with this…” And Larry said, “We are not really interested in search. We are making an AI.” So from the very beginning the mission for Google was not to use AI to make their search better, but to use search to make an AI.