That's how my chance to perform bypass surgery on Google's still-beating heart came about. My comrades and I would be disconnecting the cables one by one and reconnecting them in tightly tied bundles running in plastic troughs along the side of the server trays instead of in front of them, making it easier to move the trays in and out of the racks. Even marketeers could use a twist-tie, so we were encouraged to get our hands dirty mucking out the server farm.
"CableFest '99 lays the groundwork for the frictionless exchange of information on a global scale and will increase the knowledge available to every sentient being on the planet," I assured my wife.
Kristen looked at me and sadly shook her head. She had a PhD in Soviet history, a job as a professor, and a very sensitive bullshit detector. She tried to be supportive, but her maternal instincts were primarily focused on the three children she now worried would see little of their father. "You took a giant pay cut, and now you're working weekends. You know, the Merc might still want you back."
Saturday morning came and I pulled into the almost empty parking lot of a large, gray, windowless edifice in Santa Clara. There was no sign in front, but it was Exodus, the co-lo that housed our data center.* I joined the movement of people straggling single file through a well-fortified security checkpoint. Marketing, finance, and facilities were all represented. Even Charlie Ayers, our newly hired chef, was there. Photo IDs were checked and badges issued. Stern warnings were given. We were not, repeat, not to touch anyone else's stuff.
And then we were in.
Unless you're a sysadmin, electrician, or NSA stenographer, you may never have been inside a server farm. Imagine an enormous, extremely well-kept zoo, with chain-link walls draped from floor to ceiling creating rows of large fenced cages vanishing somewhere in the far, dark reaches of the Matrix. Inside each cage is a mammoth case (or several mammoth cases) constructed of stylish black metal and glass, crouched on a raised white-tile floor into which cables dive and resurface like dolphins. Glowing green and red lights flicker as disks whir, whistle, and stop, but no human voices are ever heard as frigid air pours out of exposed ceiling vents and splashes against shiny surfaces and around hard edges.
The overwhelming impression, as Jim led us past cage after cage of cooled processing power, was of fetishistic efficiency. Clean, pristine, and smoothly sculpted, these were more than machines, they were totems of the Internet economy. Here was eBay. Here Yahoo. Here Inktomi. Welcome to Stonehenge for the Information Age.
The common design element seemed to be a mechanized monolith centered in each cage, surrounded by ample space to set up a desk and a few chairs, with enough room left over for a small party of proto-humans to dance about beating their chests and throwing slide rules into the air.
At last we arrived at Google's cage. Less than six hundred square feet, it felt like a shotgun shack blighting a neighborhood of gated mansions. Every square inch was crammed with racks bristling with stripped-down CPUs. There were twenty-one racks and more than fifteen hundred machines, each sprouting cables like Play-Doh pushed through a spaghetti press. Where other cages were right-angled and inorganic, Google's swarmed with life, a giant termite mound dense with frenetic activity and intersecting curves. Narrow aisles ran between the rows of cabinets, providing barely enough space to pass if you didn't mind shredding clothes and skin on projecting screws and metal shards.
It was improbably hot after our stroll through a freezer to get there, and we were soon sweating and shedding outerwear. On the floor, sixteen-inch metal fans vibrated and vainly pushed back against the heat seeping out from the racks around us—their feeble force doing little more than raise the temperature of Inktomi's adjacent cage by a few degrees.
We went to work. First the ops team attached Panduit cable troughs to the sides of the cabinets with adhesive tape. Then we began gently placing the free-hanging cables in the troughs and twist-tying them together so they no longer draped over the face of the machines like the bangs of a Harajuku Girl.
I tackled the rack labeled "U." It has long since been retired, but I like to think that those user queries routed to U got their answers a nanosecond or two faster because of my careful combing of the cables.
Why, you might ask, did Google do things this way? In addition to the efficiency gained by running cheap, redundant servers, Google was exploiting a loophole in the laws of co-lo economics. Exodus, like most hosting centers, charged tenants by the square foot. So Inktomi paid the same amount for hosting fifty servers as Google paid for hosting fifteen hundred. And the kicker? Power, which becomes surprisingly expensive when you gulp enough to light a neighborhood, was included in the rent. When Urs renegotiated the lease with Exodus, Jim spelled out exactly how much power he needed. Not the eight twenty-amp circuits normally allocated to a cage the size of Google's; he wanted fifty-six.
"You just want that in case there's a spike, right?" asked the Exodus sales rep with a look of surprise. "There's no way you really need that much power for a cage that size."
"No," Jim told him. "I really need all fifty-six to run our machines."
It's rumored that at one point Google's power consumption exceeded Exodus's projections fifty times over.* It didn't help that Google sometimes started all of its machines at once, which blew circuit breakers left and right until Google instituted five-second delays to keep from burning down the house.
Air-conditioning came standard, too. Again, Exodus based their calculations on a reasonability curve. No reasonable company would cram fifteen hundred micro-blast furnaces into a single cage, because that would require installing a separate A/C unit. Google did. We were a high-maintenance client.
CableFest '99 was the one and only time I entered a Google data center. It gave me an appreciation of the magnitude of what we were building and how differently we were doing it. I can't say it inspired confidence to lay my untrained hands on our cheap little generic servers, lying open to the controlled elements on crumbly corkboards, while next door, Inktomi's high priests tended to sleek state-of-the-art machines that loomed like the Death Star. But the arrangement seemed to work pretty well for us, and I decided not to worry about things that were beyond my ken.
Very smart people were obsessing about the viability of Google's back end, and unbeknownst to me, I would soon be obsessing about the viability of my own.
Meet the Marketers
"Once she had accomplished that," Cindy was explaining to our small marketing team, "she had the world by its oyster."
I smiled. New fodder for the quote board I'd pinned up on my cubicle wall, which still featured Cindy's last pronouncement, "That's what happens when that happens."
Our department consisted of a small cadre with mixed levels of experience in marketing. Cindy was the boss and acting VP. She was close to my age, very funny (usually intentionally), and always in a hurry, which led to an alarming number of emails in which her fingers failed to keep up with her thoughts. She had started as a print journalist, then done PR duty under some of the most notorious tycoons in the Valley, where she had become personally acquainted with every reporter who talked or typed about technology. She focused on public relations, which Larry and Sergey supported as the most cost-effective way to promote the company.
Cindy exuded a wholesome Laura Petrie vibe that I found comforting, and I felt a connection with her because of our common history at newspapers. As she bounced around the department, a whirling dynamo of positive energy, she urged us to take risks, try new things, and let nothing stand in our way. We started referring to her as "Small. But mighty." Those qualities cut both ways.
"Larry and Sergey were always skeptical about traditional marketing," Cindy recalls. "They wanted Google to stand apart from others by not doing what everyone else was doing ... Let the other guys with inferior products blow their budgets on noise-making, while we stayed focused on building a better mousetrap." That skepticism translated into constant questioning about everything marketing proposed. The department only exi
sted because someone (a board member or a friend from Stanford) had insisted the founders needed people to do all the stuff that wasn't engineering.
Cindy pushed back against the constant pressure to prove her department was not a waste of payroll, but she also let us know that expectations were high. When we performed below her professional standards, she rebuked us for "Mickey Mouse behavior" with an intensity as devastating and unexpected as the tornados that swept her native Nebraska. I learned to keep an eye out for storm warnings.
My counterpart on the offline branding side of things was Shari Fujii, a thin, thoughtful, hyperkinetic marketing professional with an MBA and a tendency to exclaim that the impact of any given action would be "huge." We often commiserated about Larry and Sergey's abysmal lack of regard for our department and its work. Coming out of a company run by journalists, I found it more of the same, but Shari struggled to make it fit with her experience at brand-driven companies, where marketing summoned the sun to begin each new day.
The other key player in my world was Karen White, the webmaster. Karen had been a casino dealer in North Dakota when she decided to teach herself the ins and outs of creating web pages. Cindy had discovered her at a previous job and brought her to Google. I soon understood why. Karen had the organizational skills and disposition of a NASA launch coordinator. Industrious, objective, unflappable, and willing to stretch her day across multiple time zones, Karen took all the words I threw together and arranged them in pretty columns on our website. She had more influence on the overall look of Google than anyone who worked on it after Larry and Sergey's original "non-design" design.
Other than Susan Wojcicki, who had put her MBA to work at Intel, our group was new to marketing. Google hired Stanford grads in bulk and set them loose in the halls. If they didn't secure a role elsewhere, they rolled downhill to our department, where the assumption seemed to be that no special skills were required.
"The founders were okay with a loose shag bag of marketing folks who were at the ready to execute on their whims," Cindy told me, "but a real marketing department with a VP, proper organization, funding, and a strategy was not a priority." As a result, our world was without form and confusion was on the faces of those who dwelled within it.
"Who's working on our letterhead?" I asked Cindy. "Who handles sponsorship requests?" Were these areas that fell into my domain? I was seeking more than organizational clarity. I wanted to be sure that there was some substance to my job, something I could cling to when people asked, as they inevitably would, "What exactly do you do here?"
"No structure, foundation, or control," is how Heather Cairns, Google's HR lead at the time, remembers the company's early days. "Even if someone had a manager, that manager was inexperienced and provided no leadership. People weren't used to authority and wouldn't adhere to it—it was a completely unmanaged workforce that was bouncing off the walls like a tornado. I didn't pretend to have any control over it ... I just went home at night to drink, thinking, 'We're gonna crash and burn.'"
Keeping It Clean
"Our site is kind of a mess," Cindy said to Karen and me my second week on the job. "Can you work up some guidelines to clean it up?"
We had no rules governing what went on Google.com. Something new launched, it got mentioned on the homepage. We won an award, that went up too. Our other pages were equally devoid of planning and design. There were job listings, some help content, contact information, and brief profiles of the executive team. As with everything else at the company, our user interface (UI: the look and feel of our website) operated on the principle that we should minimize the time it took for users to find what they wanted.
Unlike Yahoo.
Yahoo's homepage had links to apparel, computers, DVDs, travel, TV listings, weather, games, yellow pages, stock quotes, and chat. It got busier with every passing day. The most prominent feature on the page was Yahoo's hand-built directory with its fourteen major categories from Arts & Humanities to Society & Culture, beneath which were links to all known points in the Dewey decimal system. Buried in the middle of all the text links was a search box powered by our nemesis Inktomi.
Inktomi hadn't always owned that space. AltaVista had provided search to Yahoo until 1998, but they made the fatal mistake of building their own portal site and stealing users from their customer (competing with your own distributor is known as "channel conflict"). Inktomi had no "consumer-facing" search site,* so they weren't Yahoo's competitors, which also gave them a clear shot at Microsoft's MSN network and America Online (AOL). Inktomi locked those customers up as well, completing their trifecta of high-traffic Internet sites and ensuring that the state of search across the web was commoditized. You could get any flavor of search you wanted, as long as it was Inktomi. They owned the search market and sat on it as fat and happy as the enormous customers they served.
Other portals wanted a piece of Yahoo's traffic: Excite, Lycos, and Disney's Go.com. And other search companies, like AlltheWeb, Teoma, and HotBot, fought alongside Google for the crumbs falling from Inktomi's table. While Wall Street focused on the portal wars, the struggle for search domination wasn't of much interest to anyone but a handful of analysts. There was no money in it. Well, not much money.
In February 1998, a small Pasadena company named GoTo started auctioning placement in search results they bought from other providers. Six months later, they claimed to have more than a thousand paying customers. According to GoTo, you didn't need fancy algorithms to determine relevance, just the invisible hand of the free market. Any company bidding for placement at the top of the results must be a good match for the term being searched. At Google, we found that concept ridiculous. Bidding-based ranking was clearly inferior to results based on an algorithm. Bidding was driven by imprecise humans. Humans bad. Math good. We knew about GoTo, but we discounted their "non-technological" approach. That proved to be unwise. We gave them a head start, and for the next four years we would fight them for supremacy in the online advertising market.
Codifying some UI guidelines† would be a good beginning project, I thought. How tough could it be to come up with some design rules for a page containing nothing but a search box, a hundred or so text characters, and some corporate shovelware behind it?* Besides, working with Karen was like drawing the right lab partner at school. Even if I screwed up, Karen wouldn't let us fail. We knocked a proposal together in less than a week.
Google was fast, accurate, and easy to use—that's what our users told us. Sergey wanted our site to be "fun" as well. Yeah, great, it's fine to have fun occasionally, but Karen and I agreed that whimsical elements shouldn't get in the way of users getting things done. We explicitly stated the obvious: "The personality of the site should under no circumstances interfere with the speed of results delivery, the accuracy of the results, or the ease in using the search functionality." An axiom we would unintentionally prove soon enough.
The rest of the proposal involved other obvious points—tweaks to what existed rather than a major overhaul—like adding decorative graphics to our corporate section. That didn't fly.
"Yahoo doesn't use images beyond the homepage," Larry reminded us, "and they have millions of users. Images take time to transfer across the Internet and slow things down." Larry and Sergey rolled on the floor rapturously speaking in tongues when we shaved a nanosecond off the time it took a page to load. Or to read. "I want all the content of the About Google section on one page," Larry said. "It would be faster to scroll up and down one page than to click from page to page and wait for it to load."
"But no one's going to scroll down a hundred pages," I said, not sure if he was joking. He wasn't, but we managed to argue him out of it. Other suggestions fell by the wayside, like a help link, a tagline, and an embarrassingly naive idea Sergey had to change the homepage logo every day to build user interest. Professional branding people were in the house now, and we would never abide such amateur antics. Overall, Larry and Sergey gave a thumbs-up, proclaiming our guidelines "sensib
le." High praise indeed.
I let out a sigh of relief. Now I got it. This was what I'd been hired to do. If I hadn't knocked my first project out of the park, at least I'd hit a solid double. Everyone seemed reasonable and receptive to new ideas, and the feedback made sense. I hadn't done anything terribly unconventional, yet my ideas had been accepted.
"Yep," I thought, "it's all going to work out just fine."
Birth of a Data Agnostic
"As of last night, Google's result font has become sans-serif," engineer Marissa Mayer announced to the company at large. "We tested the change and Larry and I reviewed it with some other engineers who were here and offered opinions about it."
I had seen Marissa's name on a note Sergey forwarded to the new marketing group a couple of days earlier. She had suggested we replace our temporary slogan—"Best Search Engine in the World. Promise"—with one Urs had come up with: "The Little Engine That Could." I didn't particularly like either line, though Marissa had constructed a detailed rationale for associating Google with the "scrappy," "determined," and ultimately "triumphant" children's book character. Besides, she pointed out, look at the importance of Ask Jeeves's tagline to their valuation.
Shari thanked Marissa and explained that we didn't have a slogan, just a phrase that was printed on some cards until we could properly research our brand character. Marketing had it under control.
Marissa, like Susan, was an old-timer who had come over from the Palo Alto office. Before that she had been a Stanford student. When I finally met her, I was struck by the intensity and scope of her interests. If everyone else at Google was a hundred-watt bulb illuminating a single corner of the company, Marissa was a flashing neon sign, casting light and shadow in all colors across the entire Googleplex. Trying to keep up with her could induce seizures. Her primary role was as a software engineer, but she was temporarily working on UI design. She had uncovered research indicating that sans-serif fonts were easier to read, so she and engineer Craig Silverstein had decided to change the results font to Verdana. It didn't happen because Karen and I had suggested a move to sans serif in our guidelines. In fact, Marissa may not have even seen our proposal.
I'm Feeling Lucky: The Confessions of Google Employee Number 59 Page 4