In The Plex

by Steve Levy

  AdWords Premium even had a way to enforce ad quality, a daily email called the underperforming keyword list. Even though Google was charging by how many people saw the ad, it tracked very closely how many people actually clicked on the ads. If the rate was under 1 percent, Google would pull the ad. “That was four times the average success rate of current ads,” says Armstrong. “So if you told anyone else in the Internet industry at that time to shut off ads with a 1 percent click rate they would have said, ‘What are you doing?’” The businesses with the underperforming ads would often go ballistic when Google told them that they had to improve the ads or find different keywords. The traditionalists would rage: Who the hell is Google to tell me the success of my ads? I’ve been in advertising for fifty years—I know what a bad ad is, and this isn’t it! “We’d say yes they are, and here’s the data,” says Armstrong. “It was a major reason that some of our people flamed out, day after day of going to advertisers who told us we were wrong.”

  The policy reflected the different philosophy Google brought to advertising in general. Google ads were answers. They were solutions. “Ideally we wanted people to have a 50 to 100 percent click rate,” says Armstrong.

  Jeff Levick, whose job it was to gin up ads from companies servicing other businesses (B2B), would cold-call prospective advertisers. They would say, “What’s Google?” He would tell them about the searches conducted in Google and what keywords were already performing for advertisers. One sector ripe for this pitch was the box business—boxes for shipping, boxes stuffed with Bubble Wrap. So he called a company called Uline, which, like Levick, was based in Chicago. “Do you know that in the last twenty-four hours 1,500 people typed the word ‘boxes’ into the Google search engine?” he said to the guy in purchasing whom he finally managed to reach. “Would you like those people to come to your website?” Levick wound up doing a lot of business in boxes.

  With Premium Sunset the algorithm was displacing the handshake. The system itself would police ad quality by estimating the success of an ad and incorporating that into the bid price. And the sales force would have different kinds of interactions with clients. The old job was making a sale. Their new job would be … getting the big companies they dealt with to place bids in an auction? “We thought it was a little half cocked,” says Jeff Levick. “If we let the auction set prices, we worried that we could actually lose a lot of money.”

  But the die was cast. Tim Armstrong, the executive in the New York office in charge of sales, gave his people an upbeat description of the system. Schmidt came to New York to assure them that it was the right move. “People were extremely upset, because this was a material change in the way they were doing business,” Schmidt would later recall. Ultimately, since the engineers in Mountain View had made good on their promises so far, the salespeople trusted them on this one. They weren’t going to be replaced. They were going to assume a new role as mediators between advertisers and algorithms.

  “Our group’s job was to build the largest bridge we could between Silicon Valley and Madison Avenue,” says Armstrong. “It was really bringing science to the art of advertising and being able to scale the art of advertising through science.”

  For Jeff Levick the big test came in his favorite product category—the boxes. Box firms had become some of the biggest advertisers at Google, and he spent a lot of time on the road seeing them, one in Southern California, one in Boston, and one nearby in Chicago. All were seeing an excellent return in their investments in Premium. Now Levick would explain that Google was pulling the plug on Premium and they were now going to have to participate in a high-stakes version of eBay. “The guy in California literally almost threw us out of his office and told us to fuck ourselves. The guy in Chicago said, ‘This is going to be the worst business move you guys ever made.’ But the guy in Massachusetts said, ‘I trust you.’”

  It wasn’t only trust that led the advertiser to stick. “The guy knew math,” says Levick. When all the numbers were crunched—and Google worked hard to give advertisers all the crunching tools they would hope for—advertisers saw that the auction system paid off for them.

  Even sectors that had once been deemed impossible proved winnable. The first time Tim Armstrong visited General Motors, in 2005, “they kicked us out of the building,” says Levick. “They said, ‘We’re never going to buy anything from you, don’t waste our time and don’t come back.’” When Google salespeople visited BMW, they got a similar reaction: Google is a fad, said the auto exec. “Who does research on cars online? They just use Consumer Reports!”

  But Google kept at it, slowly collecting people who weren’t fossils, and eventually Jeff Levick was invited to represent Google at a GM global marketing event. His presentation underlined the fact that 80 percent of car buyers do research their purchase online, and almost all of them use Google to do it. In Mexico, for example, Google had 90 percent of the search market and millions of auto-related search queries—yet GM spent only 1 percent of its ad budget on online marketing. Even Rick Wagoner, the company’s CEO, was sentient enough to see the absurdity of it.

  Google had tools to help advertisers, but they were rudimentary. Salar Kamangar tapped a smart young associate product manager named Wesley Chan to improve the services. One of Google’s better tools was called conversion tracking, which made rough estimates of how many users were lured by AdWords to the checkout page on a website, but “it was miserable,” says Chan. It was hard to set up and not very accurate. A number of independent companies had sprung up to provide analytic services, but Chan found most of them cumbersome. “You pay $5,000 or $10,000 a month plus the consultant services, and it’s still hard to read the reports.”

  Chan decided that Google needed a new product that would deliver a much higher level of service—something that would give a full reporting of all sorts of information about a website, including how many people visited it, which sites referred them, and of course whether the visitors from ad networks such as AdWords actually bought something. But he didn’t have many engineers at his disposal. “So I decided, ‘I’m going to buy something,’ even though I’d never bought a company before in my life.”

  He quickly learned how. First, scan the marketplace until you found a match. In this case it was a small firm called Urchin Software, which offered a better quality of analytics and was run by guys who seemed Googley. Then propose a partnership, because any company worth buying really doesn’t want to sell itself. Finally, switch the rules and ask the founders if they want to join Google. All along, you had to operate a second front—getting the Google brain trust to okay the purchase. In this case, Larry Page was skeptical, but Chan won him over. After months of negotiation, Google bought Urchin for about $20 million in late 2004.

  Thus began a long process of making Urchin into what became known as Google Analytics. Chan’s original idea was that Google would charge $500 a month to use the service, but offer discounts to AdWords customers. But Chan’s team was undermanned and had no experience in building a billing system. Finally, he went to Page and suggested that Google offer the product for free. It would take another eighteen months to build a billing system, and wasn’t it better to spend all that energy figuring out ways to make users happy? Page relented, and in November 2005, Google Analytics went live.

  Chan had predicted that opening up this easy-to-use service that would provide for instant statistics on websites—free—could result in ten times the current activity in analytical products. So he “provisioned” the data centers to handle the volume. (This meant reserving the necessary clusters of servers to handle the estimated load of a service.) Nonetheless, within forty-eight hours, virtually all of Google’s servers crashed, unable to process the tidal wave of data washing into the company’s servers. Eric Schmidt would later call the meltdown Google’s most successful disaster. For almost a year, Google had to limit access until finally opening the service up to all comers. Even though Google Analytics didn’t require a client to be an AdWord
s customer, the data it provided revealed the value of the Google ad world, enticed new customers, and kept current ones assured that their investment in Google ads was a genius purchase. “Analytics generates about three billion dollars in extra revenue,” says Chan. “Know more, spend more.”

  “Every advertising should be measurable,” says Susan Wojcicki. “You should be able to adjust it, right? Then you should be able to tune it, track the right users, and target it to the right people.”

  Eric Schmidt saw this dynamic in action even before Analytics was rolled out, on the day that the sun set on AdWords Premium. Schmidt had come to New York to witness the historic switch. At around five o’clock he was sitting in a cubicle and couldn’t help but overhear a conversation being conducted between a young woman in Google’s sales force and a client on the other end of the phone. She seemed typical of the people there: dark-haired, cut-to-the-chase, loud in a way that shouted “New York.” Maybe not so Googley. She was explaining the transition to a baffled client. It was clearly a difficult conversation. Afterward, Schmidt introduced himself and apologized for the trouble that the transition was causing her. She explained to him that the client’s tension was rooted in the fact that Google ads were the way his company made all its money.

  “You’re kidding,” said Schmidt. She wasn’t.

  Schmidt finally got it. He’d been viewing the transformation of the advertising business from thirty thousand feet, but now he saw firsthand that countless businesses had discarded the old handshake method of buying ads and had embraced Google’s model. “Our system doesn’t work that way,” Schmidt says. “There’s an auction, it sets the price, you win, it’s a fair price, and then there’s another auction.” The role of Google’s saleswoman was not to sell her client something he didn’t want, but to provide data to help him sell more, using tools that Google provided not only to assess the ad but possibly to transform the way his company thought about itself. Not to mention the transformation of the ad industry, which could never again claim that its business was an unquantifiable mystery. The right algorithm would make partners of the woman and her client, make everything efficient and measurable, and turn on the money tap for both sides. And since Google had devised the best algorithms, it had emerged as the winner of the ad game. The next step was to leverage that advantage so that no one else could ever come close.

  Premium Sunset was an apotheosis for Google. Google’s business plans may have begun as a means to support the search business, where its founders’ hearts lay. But by the mid-2000s, Google’s business became much more. In most advertising-driven companies, the business side was regarded as less interesting and creative than the consumer-directed activities. But at Google, the ad effort became a more or less equal sibling to search. When Google recruited its alpha geeks, it was just as likely to ask them to get involved in some AdWords project as it was to ask them to focus on some effort in search or apps. The reason was that in order to succeed on a major scale, AdWords needed that kind of talent in mathematics, computer science, and statistics.

  “Search has a luxury that ads don’t have,” says Jeff Huber, who came to Google to head engineering on the ad side in 2003. Previously, he’d been eBay’s vice president of architecture and systems development. “Yes, search is a huge system, but it’s stateless—you can easily serve it from ten different places in the world, and if this version is slightly different than that version, the user won’t know, nobody will notice. But with advertising, the state is important, because advertisers are always updating their campaigns, and microtransactions are happening at ferocious rates per second, and all that has to be synchronized.” Compared to Google’s demands, the auction volume that Huber handled at eBay was like spitting in the ocean—and this complication of “state” meant technical challenges that would keep brilliant computer scientists up at night. “We needed to invest. The amount of data was doubling every quarter. Things were straining at the seams, and we would have ad outages or delays of stats reporting of a day or more. Every time we had an operational issue, it became national news. There were very explicit discussions about how we were going to survive Christmas in 2004.”

  That was just the operations end of it, where Huber had to hire (or lure from other areas of Google) engineers and computer scientists to scale the system and build new infrastructure. An even tougher part of the system was performing the complex calculations that kept the system vital. Serious math and statistics were required. In order to figure out the critical ad quality score, Google had to estimate in advance how many users would click on an ad. That involved building systems that could process an incredible amount of data and accurately predict a future event millions of times a day. Since the Google ad model depended on absolute mastery in predicting click-through, over the years the company would spend enormous amounts of effort and prodigious amounts of brainpower to get it right.

  A new arrival at Google would act as a godfather to the advertising effort. His name was Hal Varian, and he would eventually hold the title of Google’s chief economist. In 2001 newly hired CEO Eric Schmidt ran into Varian at the Aspen Institute, Schmidt was with Larry Page, and Varian remembers thinking, Why did Eric bring his nephew from high school here? Nonetheless, Schmidt, whose father was an economist, suggested to Varian that he spend time at Google, maybe a day or two a week. On his first visit Varian asked Schmidt what he might do. “Why don’t you take a look at the ad auction?” Schmidt told him. “It might make us a little money.”

  Varian was uniquely qualified to vet Google’s approach to making money online. He’d been thinking like an economist ever since he was twelve, when he’d read Isaac Asimov’s Foundation Trilogy and become enchanted with a character who constructed mathematical models to explain societal behavior. “When I went to college at MIT, I looked around for that subject,” he says. “I thought it might have been psychology or sociology, but it was economics.” He also learned to program computers at MIT. After getting his doctorate at Berkeley, he taught at MIT and then at the University of Michigan, where he began studying the topology of the Internet from an economic perspective. He became fascinated with what seemed to him “a lab experiment that got loose—it wasn’t designed for commerce at all.” But Varian understood that the net’s unique attributes gave it an opportunity to redefine commerce, and he took that idea with him to Berkeley in the mid-1990s, when he became dean of the UC Berkeley School of Information Management. With Carl Shapiro, he wrote a popular book called Information Rules: A Strategic Guide to the Network Economy, and became the go-to economist on e-commerce.

  After examining Google’s system, Varian realized that it was the embodiment of the Silicon Valley ethic he’d been studying. Though the Internet was different from other media, most Internet companies were still selling ads the way Madison Avenue had always done it. Google saw the entire exchange differently. Advertising in Google was less comparable to television or print than it was to computer dating. Google was a yenta—the Yiddish term for the pesky, persistent matchmakers who linked brides and grooms in the shtetl. It matched advertisers with users. And since, as Varian says, “in economics there’s no shortage of theories,” there already was a body of work dealing with these things. One of the classic papers in the field was a 1983 study by the Harvard economist Herman Leonard that dealt with matching problems such as assigning students to dorm rooms. It was called a two-sided matching market. “Ironically, the mathematical structure of the Google auction is the same as one of those two-sided matching markets,” says Varian.

  During Varian’s first summer at Google, when he was coming in a day or two a week, he tapped a recently hired computer scientist and mathematician from Stanford named Diane Tang to create Google’s search-word advertising equivalent of the stock market, called the Keyword Pricing Index. “It’s like a consumer price index,” says Tang, who came to be known internally as the Queen of Clicks. “But instead of a basket of goods like diapers and beer and doughnuts, we have keywords.�
� Different categories were ranked by the cost per click that advertisers generally have to pay and then separated into high-cap, midcap, and low-cap bundles. “The high caps were very competitive keywords like flowers and hotels,” Tang says. (The very highest CPCs [cost-per-clicks] were for categories such as mesothelioma, used by litigation attorneys to troll for clients—winning bids could go for fifty dollars per click. Also, anything touching on insurance rates made for pricey keywords.) In the midcap realm were keywords that might vary seasonally—in the winter the price to place ads alongside results for “snowboarding” would skyrocket. Low-caps were the stuff of long tails. Meanwhile, Google had an equivalent to the Dow Jones Industrial Average: the average cost per click, which was calculated by summing up all the ad revenue and dividing by the total of paid clicks. “If you change the mix or get more low-cap ads, it can go down even though your pricing is going quite well,” says Tang.

  Tang’s goal was to construct what she calls a “data warehouse” so that the simpler analyses could be turned over to the sales force or the customers themselves—to whom Google would supply all sorts of tools to figure out where their ads were and how they were working. Meanwhile, Google collected a phalanx of statisticians, physicists, and data miners to unearth every twist and turn in the Google economy.

  “We have Hal Varian, and we have the physicists,” says Eric Schmidt. “Hal’s interaction with his group is like a professor and his students. His job is to get them to deeply understand an issue and then move it forward. And the physicists’ job is to figure out the lifetime flow of a click.”

  Varian referred to his team as “econometricians.” “Sort of a cross between statisticians and economists,” he says. Of the early statisticians hired, Daryl Pregibon joined Google in 2004, after twenty-three years as a top scientist at Bell and AT&T labs. “We needed a class of mathematical types that had a rich tool set for looking for signals in noise,” he says. “The rough rule of thumb here is one statistician for every hundred computer scientists,” he says.