Ray took unauthorized automated queries very personally. If he could figure out the spammer's email address, he sent a terse cease-and-desist warning. If he couldn't find an email address, he blocked the spammer's IP (internet protocol) address—the unique number assigned to a computer connected to the Internet—from accessing Google altogether.
No one was immune. When a user left a book on the Enter key and sent the same query to Google thirty-nine thousand times, Ray cut off access for everyone at that address. The query was "This is the CIA," and it came from that agency's headquarters in Langley, Virginia. Another user searched for "net oil importers" over and over and over again. Ray got annoyed and shut off the State Department as well.
If Ray couldn't identify a specific IP address, he contacted the spammer's Internet service provider (ISP) and asked that they track down the offender themselves and sever his access to Google. If the ISP refused to play along, Ray upped the ante—he blocked access to Google from all of the ISP's addresses. That usually got their attention. It was how Ray shut down access to Google for most of France. The French ISP definitely noticed, all the more so because at the time they were negotiating to become one of our larger customers.
Ray didn't hate the French. He did the same thing to the Germans. Also to a major American ISP, though he did post a note to their customers who complained. "The short story here," he wrote, "is that some user at your ISP was abusing Google. We were unfortunately unable to turn off access just for this evil individual. Since your ISP didn't respond to us, we had no choice but to shut off access to Google from a large number of IP addresses."
Cindy was, as she put it, "displeased" when she read Ray's note reprinted in a headline article on CNET describing Google's rude treatment of users. She "suggested" that I take over user communication related to service interruptions and "work with" Ray to smooth out the rougher edges in his correspondence. It was hard for me to keep up, because Ray was all about initiative. He was not part of the company's business-development team according to the org chart, but he never let reporting lines fence him in.
"Well, to be blunt," Ray told a partner who wanted to renew a deal for Google technology, "it's clear to us what you get out of our relationship, but it's far from clear to us that we get anything out of it. Given that, it seems like poor business practice for us to continue with it. So, unless I'm missing some key observation here, please stop performing Google searches immediately."
Many things made Ray wroth. He sent out long notes to all Googlers demanding we clean up the kitchens, the locker rooms, our interviewing techniques, our security practices, our personal habits, and our grammar. He also urged us to recycle our trash at every opportunity. Once a burr got under his saddle, he didn't wait for it to work itself out. "Can we please, please, please finally just end our relationship with these leeches?" he begged of Larry Page when another partner continued to annoy him. "If only to make me happy?"
Impulsive and opinionated, Ray will always personify for me Google's engineering id, a lone cowboy patrolling the electronic frontier in shocking-pink shorts, facing down the black hats and making them blink, then riding off into a sunset that was only half as colorful as he was.
A single engineer holding that kind of power speaks to the assumptions inherent in Google's culture. Individuals were considered capable of weighing the effects of their actions and presumed to have the best interests of the company (and Google's users) at heart. We were encouraged to act on those interests without hesitation. Spend time doing, not deciding.
Of all the elements of "big-company thinking" I had to unlearn, that was one of the hardest. I constantly sought reassurance that I was empowered to move to the next step, only to be asked, "Why haven't you finished that already?" The upside of this philosophy is that Google did things quickly, most of which turned out to be positive. The downside is that individual Googlers sometimes misinterpreted exactly how much power they possessed and when it was okay to use it.
Shari had discovered the downside the hard way. She had reached the breaking point with Larry and Sergey. They weren't supporting her work with the promotion agency, and without outside help, she couldn't move forward. She threw up her hands in frustration, and while they were up, she tossed in the towel.
At her farewell party at a local Mexican restaurant, I said goodbye to the one other person at Google who completely understood the practice of branding for customer acquisition. Google didn't do that kind of marketing. The company rejected any attempt to graft traditional practices onto its new breed of business.
Over salsa and Dos Equis toasts, I resolved that I would remain open to new ideas and new approaches. I would make it work. I would prove to myself and to my ever-adaptable colleagues that this old Doug could still learn new tricks.
PART II
GOOGLE GROWS AND FINDS ITS VOICE
Beyond a startup.
Not yet a search behemoth.
Google's awkward phase.
Chapter 11
Liftoff
WHILE I HAD been trying to figure out what to do next, the engineers had been killing themselves to do the big, hard, complicated things that absolutely needed to be done. Their yearlong effort would come to fruition just about the time I started to get my bearings.
The engineering story began in June 1999—before I had even heard of Google. Jim Reese, the neurosurgeon turned sysadmin, had just been hired. On his first day, he arrived at eight a.m. and worked straight through for fourteen hours. The next day he came in a little later—about ten a.m.—to add backup servers to Google's intranet and to handle networking issues in the Plex. He left the office around four for an early dinner on his way to Exodus, Google's data center, where he stayed until five in the morning. He did the same the next day, and again every day that week, including Saturday and Sunday. His task, assigned by Larry without explanation, was to install two thousand new servers and bring them online.
That many computers wouldn't fit in the cage Google owned at the time, so Jim needed to arrange for additional space at the data center. "I worked as hard as I could," he said, "negotiating with facilities at Exodus. In 1999, cage space was hard to come by and Exodus was pretty full." Partly that was because of companies like eBay, whose cage was near Google's. "They had a cage ten or twenty times our size and they had perhaps eighty computers in it," Jim recalls, "whereas we had eighty computers in one rack." There were nine racks crowded into Google's cage—but, as Jim and his new associate Schwim realized when they looked closely, it wasn't at capacity.
"If we move every cabinet on this side of the cage three inches, we'll have exactly enough room to fit in another rack," Schwim pointed out, turning sideways to squeeze down the aisle. "The only problem is, there's no way to roll a rack through here. We'll have to take off the side cage wall." They called the facilities manager, described what they wanted, and left for lunch.
When they came back, the black chain-link fencing that had protected the side of Google's space had been unbolted and removed. Jim and Schwim slid the rack in, cabled the computers, connected them to the main switch with fiber, and flipped the switch. Everything lit up the way it was supposed to. Jim double-checked the connections at the back of the rack as Schwim stood at the front typing in commands to monitor its progress.
"It's a go!" Jim heard Schwim announce from the front of the rack.
"The next thing I knew," Jim recalls, "I'm sitting on my butt on the floor of the data center."
"We just lost the rack!" Schwim yelled. "What's going on?" He stepped around to the back and found Jim on his back, groggily rubbing the crown of his head, a two-hundred-pound metal crossbeam, smeared with blood, lying beside him where it had fallen from the top of the cage.
"Uh...," said Jim, shakily pointing upward toward a batch of severed cables that had lain in the beam's path, "we're going to need more fiber."
"You're the only neurosurgeon around," replied Schwim, assessing the situation with both concern and an enginee
r's practicality. "Do you think you can fix yourself?"
The facilities team, for reasons known only to them, had unscrewed the support for the crossbeam while removing the wall. In Jim's professional medical opinion, the beam would have done him considerably more damage if it had landed a couple of inches to either side. "They were very, very, very kind to us after that," he said about the Exodus crew.
Jim finally found some unoccupied space in a corner of the building and Exodus agreed to throw fence walls around it. He spent the better part of June and July installing two thousand brand-new computers into the cage. The machines didn't always work. They were built quickly and with parts purchased at very reasonable prices. "For some racks," Jim recalls, "we got fifty-six out of eighty working, so we'd spend a week installing these machines and then another week repairing the ones that didn't install."
Eventually Larry let Jim in on why his work was so urgent. Google had signed a deal with Netscape to be their fall-through search engine. If Netscape users couldn't find what they wanted using Netscape's open directory, they would be able to search from the directory page using Google. So Google needed more computing power to handle the potential traffic.
Jim load-tested Google's capacity as he added machines—checking to see that it could handle the increased traffic and any occasional spikes that might occur. "In general," he told me, "you like to see two times capacity. For peaks, you like to see four times capacity. Netscape anticipated a one-point-seven-times increase over existing Google traffic, so I tested it. At nearly five times, we were completely in the clear."
It wasn't easy. Jim and Schwim were still at the data center installing machines the night of June 24. Netscape would announce the deal and start directing traffic to Google the morning of June 25. Schwim worked until two a.m., when the cumulative lack of sleep caught up with him and he went home to crash.
Fortunately, Jim had recruited another tech guy to help them over the finish line. Though it had been a while since Sergey had dirtied his hands installing machines, he stayed at the data center with Jim until five a.m. "He didn't know all the technical details of how the routing went," Jim remembered, "but he was in there crawling under the floorboards, running cables, and hooking up switches."
For Jim it was the culmination of weeks of exhausting physical labor, and when he finally dragged himself off to bed it was with a sense of accomplishment. Google had averted a potential disaster by tripling its capacity in record time.
An hour later, his phone rang. It was Sergey. "Get in here right away. We're melting down."
Netscape's press release had hit the newswires at six a.m. West Coast time. Within seconds Google's traffic had increased not the expected one-point-seven times, but sevenfold. The servers couldn't handle the load. Sergey and Jim rushed back to Exodus and began desperately throwing the last batch of machines they had into racks and hooking them up.
Meanwhile they did everything they could to clear away extraneous demands on Google's infrastructure. They stopped the crawler from adding websites to Google's index and reallocated those machines to serving results. It helped, but not enough. Response times had slowed perceptibly, and some users got no results at all. Google's most important launch to date teetered on the brink of becoming an epic pooch-screwing.
The atmosphere in the office Craig Silverstein shared with Amit Patel was grim as Larry and Sergey, Urs, and the rest of Google's engineers reviewed their options. Netscape was not a small partner like their first client, VMWare. If this relationship went down the tubes, everyone would know and Google's tech reputation would be toast. They could think of only one way to increase capacity to handle Netscape's users.
"Shut off queries to Google.com," Larry instructed the team.
For the next couple of hours, anyone who went to Google.com saw a static page explaining that Google was down. Every computer and every bit of bandwidth Google had at its command was serving results to Netscape users. Larry and Sergey were risking their own site's reputation to maintain credibility as a reliable technology partner.
By lunchtime, traffic had subsided enough that Larry and Sergey gave the okay to turn Google.com back on. Schwim and Jim returned to Exodus to finish installing the last of the servers, and within four hours they had brought an additional three hundred machines online, ending the immediate crisis.
As Jim and Schwim left the controlled environment of the data center and headed out into the warm evening air, they received another call. Netscape's engineering team was at the Tied House Brewery in Mountain View, celebrating the partnership, and they wanted Google's tech team to join them.
"They threw us a great post-launch party," Jim remembers. "And the thing that came up over and over again was, 'I can't believe you guys shut down your own site just to serve our traffic.'" The Googlers in attendance noted well that their sacrifice had paid off handsomely. The deal with Netscape promised to blossom into a beautiful friendship. Google gained not only trust, but also access to a whole new set of data in Netscape's query stream—data we could analyze and compare with our own traffic. Most important, the company's first major crisis battle-hardened it. Larry and Sergey would never again underestimate the challenges of occupying new territory. Though it seemed epic at the time, the battle of Netscape would go down as a minor skirmish once Google fully engaged the major players in the war for search supremacy.
That day was coming.
What's Going Down?
A little after midnight one Saturday night in the fall of 1999, Jim's phone interrupted his sleep again. Again it was Sergey.
"The site's down. What's up?" he wanted to know.
"Not me," Jim replied with a yawn. "You woke me."
A circuit breaker at Exodus had flipped, taking down Google's main switch, an inexpensive little piece of Hewlett-Packard hardware through which all of Google's traffic flowed. Exodus had set up the switch before Google moved the first racks into its cage, and had done it in a hurry. The device had been placed on the floor under one of the racks and was cabled in such a way that it had to stay there. It was known to all the techs by the designation "Switch on the ground." There was no backup, and when it crashed Google went offline until someone did something about it.
"Sergey had been at a party. He came home and noticed we were down," recalls Jim, who logged in, figured out the problem, and had Exodus turn the circuit on again. Google was offline for about half an hour.
"We should probably be monitoring our site, huh?" said Sergey when Jim called to let him know it was back up.
Jim spent the rest of Saturday night and Sunday morning writing a script to monitor Google. His script checked the site every five seconds to make sure it was operational and called a phone number if something went wrong. The next week everyone in operations got a pager.
Google had gone dark for a second time, but no tempers flared and no heads rolled. "If Larry and Sergey were upset about anything," Jim told me, "it was, Why didn't any of us think of that? We're a bunch of bright people here and none of us even thought to monitor our own site."
The pager alert system created problems of its own. "Claus,"* a logs engineer, was one of the first to be hooked up, and he watched carefully as our traffic numbers kept redlining, threatening to crash the logs system. The logs were money—we billed advertisers on the basis of the data they contained—so he set up his own scripts to crunch the numbers and to call his pager when they were done. That happened about three times an hour, every hour, all day long. According to engineer Chad Lester, Claus "kept Google alive in the early days. He'd be sleeping at his desk in twenty-minute intervals between pages. One month he got a pager bill in the thousands of dollars."
Google renegotiated its pager service contract but never compromised on-site reliability again. Google.com would stay online, no matter what.
Here Comes You-Know-Who
During the spring of 2000, I didn't sense any great strain in the fabric of the company as I grew accustomed to its rhythms. The basi
c elements had coalesced: a physical plant, a core engineering team, finance and HR staff, and even marketing in support of a product for which the demand seemed insatiable. The coming months would be about holding on. A previously unmet need was rushing headlong toward the provider of a free solution—bucking our audience numbers higher and higher with each lunge forward. On May 8, 2000, Google's traffic topped eight million searches a day. Two weeks later, it was nine million. In theory, we could grow forever, but each bounding leap threatened to bring our ride to an abrupt and messy end because we couldn't add capacity fast enough.
The biggest jump lay just weeks ahead. No one spoke about it, but as I stood in line at the café, debating what I could actually eat from Charlie's Appalachian Day menu (pickled pigs feet, okra consommé, free-range pork rinds, moon pies with mayonnaise, and Twinkie cheez-dogs), it seemed there were more than the usual number of empty seats. The few engineers I did glimpse hurriedly filled their trays and headed back to their desks wearing stress and fatigue like battle-tattered hockey jerseys.
Rumors and whispers about a big hairy deal had been spreading over the cables and through the cubicles, but no one would confirm whose business we were attempting to capture.
Urs knew. He rode herd on his ops team to build capacity in the data-center cages as fast as humanly possible. We would need every server we could cobble together to feed the ravenous behemoth we hoped to contain there.
We were going after Yahoo.
Inktomi's contract to supply search results to Yahoo was up for renewal in June 2000, and Yahoo did not intend to extend the partnership, a fact they were hiding from the world at large. They wanted Google to provide the fall-through search on their site, just as we did for Netscape. If users couldn't find what they wanted in Yahoo's directory, they would use Google to search the web.
I'm Feeling Lucky: The Confessions of Google Employee Number 59 Page 19