Viral Loop

by ADAM L PENENBERG

  Twitter code, built in Ruby on Rails, an out-of-the-box software framework, offered myriad benefits. It was relatively easy to create a workable prototype—in 140 characters or less you are asked to respond to the question: What are you doing?—and it’s straightforward to add additional applications. But it wasn’t designed for a mass audience. “We were taken by surprise by the scaling issue,” admits Biz Stone, one of Twitter’s founders. He points out that Twitter was first a project, a cool idea that blossomed into a real company, which counted more than 6 million users by April 2009.

  Union Square Ventures partner Fred Wilson, who invested in Twitter, believes the microblogging outfit was simply too slow in coming to grips with the severity of the scaling issues. “They’re playing catch-up now and it’s been unnecessarily difficult for them,” he says. “But Twitter is what it is because of the people who created Twitter. You could have taken a bunch of people who were the best scaling engineers and put them in a room and they wouldn’t have created Twitter. In the best situation, the people who had created Twitter would have figured out the scaling issues and found some great scaling engineers and recruited them onto the team.” But top engineers are in short supply and if you were responsible for helping Google scale to 1 zillion queries per second, you’re not going to work for Twitter. You’ll stay at Google or build your own company. Nevertheless Twitter has gradually improved its dependability. The company experienced 84 percent of its downtime in the first half of 2008, while there were no major outages in the second half.

  Some companies limit growth so their systems can handle the exponential rates that come with viral growth. Paul Buchheit, the brains behind Gmail, purposely controlled the rate of adoption by instituting an invitation-only sign-up procedure. Because Gmail offered 1,000 megabytes of storage while others gave users only 4 megabytes, Buchheit chose to drag down Gmail’s growth rate so Google could keep the application operational without risking sluggish download times, outages, data loss, or any other performance problem that often emerges with rapid scaling. After all, being able to hit the brake in a speeding car is just as important as putting your foot on the accelerator.

  Even then, it’s rarely a smooth ride. Even with controlled growth, Gmail coughed up hairballs as it grew. So did Facebook. YouTube has gone dark and so has Ning—and all of them were started by people with programming backgrounds.

  [ TECH SAVIOR ]

  As for eBay, which in 1999 was on the verge of losing everything because it couldn’t keep the site up, it would need to find someone with the engineering chops to do something that had never been done before: scale a company’s infrastructure faster than Moore’s law. For almost fifty years Moore’s law, named for Gordon Moore of Intel, accurately described the long-term trajectory of computer hardware: the number of transistors on a chip will double about every two years. What that means in real life is that the speed and performance of microchips—the brains of computer hardware—doubles over that same time period while shrinking in size by half. In the case of eBay its existing hardware was no match for its fast-growing needs. It already ran on the largest mainframe Sun Microsystems had to offer and its database servers were approaching their limits of physical growth.

  Meanwhile, eBay’s legions of users were turning on the company and migrating to other auction sites and the market punished eBay stock. When engineers finally got the site up after twenty-two hours of darkness, eBay shares, which were $180 when the last auction went through, had fallen almost $50 a share to $136 by the time the site came back, slicing $5 billion in market capitalization. Referring to the fiasco as eBay’s “near-death experience,” CEO Whitman ignored calls for her discharge and ordered her four hundred employees to phone users to apologize for the inconvenience. “It humbled the company,” Whitman told USA Today. “We were on a rocket ship…. It really stopped any idea of ‘Gee, aren’t we special,’ which was really good culturally.”

  The company had been so focused on growth and user experience that it had neglected its infrastructure. Not only was it not sturdy enough to handle its needs, there were no redundancies—no data backups—built into the network. Now that the site was limping along with scattered brownouts and glitches, a sleep-deprived Whitman, determined to prevent anything like this from ever happening again, sought a new chief technology officer. A headhunter remarked that there were perhaps ten people in the world who could get eBay out of this mess, but she had just the man: the aptly named Maynard Webb, chief information officer at Gateway, the PC maker.

  In some ways the system that Webb oversaw at Gateway was even more complex than eBay. It had to be able to configure a PC made up of two thousand separate components, and then produce and ship twenty-five thousand orders a day, an infrastructure that strung together manufacturing systems, shop-floor automation, and supply-chain management. Designing and maintaining a website and back end to handle a product that each customer could configure to taste was a major engineering feat. It had to be ready to run at all times and track an order through the system. But there were only so many stages in the shipping process and Gateway could create automatic processes to handle them and give the appearance they were working in real time when they really weren’t. While eBay’s transactions might have been simpler than Gateway’s, it had a far greater volume of interactions that came in bursts. The site had to run in real time with no margin for error. With an auction, a user wanted to know every second when somebody had outbid her. Those had to be true real-time interactions, multiplied by hundreds of millions of users. Adding to the infrastructure’s burden was the fact that eBay was doubling in size every six months, on pace to negotiate more daily transactions than NASDAQ.

  Whitman wooed Webb with the promise that he could have any and all resources at his disposal to right eBay’s ship. She was also willing to pay him more than she was earning: a salary of $450,000 (more than double hers) plus a signing bonus of $108,000, an extra $300,000 if all went well and options to buy half a million shares of eBay stock. While he was house-hunting with his wife three weeks before his start date, the site went down again. Webb left his wife to do the negotiating and jumped into the fray. He ducked an encampment of reporters who were camped outside eBay’s San Jose headquarters to reach the engineers who were furiously searching for a diagnosis. Thirteen hours later, Webb and his engineers learned what was ailing them: a software glitch.

  A sound infrastructure for a growing e-commerce company should be flexible, scalable, and reliable. EBay’s was none of these. The network was a mishmash of Solaris and Microsoft NT servers with numerous single points of failure. If a component burned out, the entire network could blow with it; if the function was tied to the central database, the outage would last for hours. Webb’s first move was to identify all single points of failure—what were the bottlenecks and how could he ease them? “Once you’re in the soup it’s hard to climb out,” Webb says. “If you fall behind the scale curve it’s very hard to get ahead with all that volume coming in.”

  Webb quickly moved to manage downtime by having his engineers construct a “warm backup”—a redundant system to take over in the event the main system went down—for each of two hundred NT servers that accepted bids and processed new members and the Sun server database that held the 3 million items listed for auction. Even if eBay experienced a systems crash, it would only be down a few hours, not days. These redundancy systems meant that if the site went down, all he would have to do was flick a switch to get back online while he figured what was ailing them. The number-one rule for any online business was to keep the lights on.

  Then he wanted to know how much wiggle room there was in the network. “What are the upper limits that the system can handle?” he asked. “When do we run out of capacity?” While his team worked on the answer, Webb identified the major bottlenecks and architectural blunders and constructed systems to handle them. His crew returned with bad news. The hardware was running at 95 percent of capacity, which meant eBay had about three w
eeks of life. Because Sun wasn’t scheduled to release a new upgrade to the server for eighteen months and eBay would soon enter the Christmas season, its busiest time of the year, Webb threw himself into scaling the architecture to remove the looming thresholds. He divided eBay’s gargantuan, nearly maxed-out database into more manageable pieces, relocating functions like accounting, customer feedback, and various product categories to separate machines. This distribution resulted in more room for growth and took away the risk of a single server dragging down the entire system.

  At times he found himself working at cross purposes, with the marketing department pushing growth strategies while Webb was trying to slow things down. “The marketing guys would want to do free listing days,” Webb recalls, “and sellers would stay up all night posting items for sale. It would drive the volume up a huge amount—a year’s worth of volume in one night. It was like “something’s gonna blow!’”

  It took time, money, and labor power to install new equipment and port over the code. In his first six months on the job Webb spent $18 million on top of the millions eBay was already budgeting for computer infrastructure. Reflecting this diversion of funds, the percentage of profit eBay accrued from listing fees fell 10 percent from earlier in the year. This led to complaints from the marketing department, which Webb shrugged off. “Demand generation is always easier than demand fulfillment,” he told them. By the end of the year most eBay users who had bailed after the blackout returned as the site became more reliable. Soon after, the company was measuring outages in mere seconds per month, as traffic and transactions continued to mushroom. And when eBay’s techies didn’t catch a glitch, its customers often would, alerting staff within seconds if something went awry. From then on, eBay touted a 99 percent uptime.

  It also continued to grow at a fierce rate. In 2001 it counted 42.4 million registered users, 95 million in 2003, and 181 million in 2005. With huge cash reserves it was able to buy two other viral loop companies: Pay-Pal for $1.5 billion in 2002 and Skype for $2.6 billion three years later. Between the three companies, eBay counted half a billion registered users by 2006. Eventually, like all viral loop companies, however, eBay hit a point of ultimate saturation. By March 2008 its revenues and profit were growing in the midteens, compared to the 30 percent growth it had grown accustomed to over the years.

  Still it remains a formidable force with a market cap of $40 billion, monthly traffic in excess of 100 million users, and 14 percent of global e-commerce revenue. It’s also a primary source of income for some 1.3 million people who are part of an entire economy that sprang up around Omidyar’s perfect market experiment.

  Perhaps if there’s one phrase that sums up eBay, it might be: “Going…going…going…still here!”

  8

  PayPal: The First Stackable Network

  Viral Synergy, Greedy Inducements, Scalable Fraud, and Battle of the Network All Stars

  One way to look at the history of the Web is to view it through the prism of viral synergy. The precipitating event was the release of the Mosaic and Netscape browsers, which encouraged the creation of sites. More sites, more reason to venture online, which yielded more sites. But this collection of seemingly disparate sites weren’t, when taken together, a single network. Nor was Hotmail, ICQ, or Napster, tools built atop the Internet’s viral plain. That had to wait for eBay, which was the first true viral network to spawn its own ecosystem, the millions of sellers who set up shop there. They in turn generated an entire economy: demand for shipping and delivery services, boxes, tape, and a transactions medium, which Pierre Omidyar’s brainchild had yet to fill—until PayPal came along.

  It was the summer of 1998 and Max Levchin, a twenty-four-year-old Ukrainian-born computer scientist from Chicago, had journeyed halfway across the country to crash on the floor of a friend’s place, wondering what to do with the rest of his life. With nothing but time on his hands, he figured he might as well check out a lecture at Stanford on political freedom and the globalization of the world economy. The guest lecturer, Peter Thiel, was a staunch libertarian who was by all accounts a genius. Levchin had always been attracted to people who might be as smart as he, and he had a personal connection to the topic. A Jew from Kiev, he had grown up in the Soviet Union where everybody was equal—except Jews, Christians, gypsies, homosexuals, and anybody who didn’t toe the Communist line. While almost everyone else in the crumbling Soviet empire lived second-class lives, the Levchins were relegated to a third-class existence, subject to harassment and denied opportunities, limited in where they could live, work, and attend school.

  Fleeing the crumbling Soviet empire most likely saved their lives. When Levchin was eleven, his mother, a physicist who worked as a government research assistant, overheard news of a leak at the Chernobyl nuclear reactor, which was on the verge of a meltdown. Acid rain misted down as the family quietly vacated their home and rushed to the train station in Kiev. After they were onboard, news of the disaster became public, and hours later, as they chugged into Crimea, Soviet guards ordered them to turn back, fearful of contamination. Following an animated discussion, Levchin’s mother convinced them to check for radiation. All were clean, except Max, whose right foot sent the Geiger counter into spasms. The guard said the boy’s bone marrow was contaminated; his leg might have to be amputated. His mother told Max to take off his shoe and he was tested again. This time he passed and they were let through, sans shoe. The culprit was a radioactive rose thorn that had lodged in his sole as they escaped Kiev.

  The family arrived in Chicago with $700 in life savings and moved in with a distant relative. Levchin taught himself English by studying reruns of Diff’rent Strokes on a busted black-and-white TV he fixed after rescuing it from the trash. His parents somehow afforded a used PC and he continued the coding and cryptography he had pursued as a preteen. After receiving his high school diploma Levchin enrolled at the University of Illinois at Champaign-Urbana. There, he heard the stories about its most famous alum—Marc Andreessen, whose company, Netscape, was taking on the world—and dreamed of following in his footsteps. Levchin earned a reputation as a brilliant and quirky engineer, a guy who never seemed to sleep, coding day and night yet still finding time to launch three start-up companies, including NetMeridian, an automated marketing-tools firm he sold to Microsoft for $100,000 just prior to graduation. While waiting for the deal to close, Levchin, practically broke, high-tailed it to Silicon Valley to figure out the rest of his life. And on a hot summer’s eve with the air static and unflinching, he met Peter Thiel, who found himself back at Stanford lecturing to the six people willing to brave the weather.

  Born in Germany and raised in California, Thiel, who was in his thirties, was once ranked among the nation’s top chess players under twenty-one. He was so hypercompetitive that after a rare loss he had angrily swept the pieces off the board. Thiel stopped playing competitively because he feared chess had become an “alternate reality in which one loses sight of the real world.” To get to the next level, he calculated, would have prevented him from attaining success in other aspects of life. Ever suspicious of crowds and groupthink, he gravitated toward libertarianism at a young age even though his parents weren’t overtly political. As a student at San Mateo High School he read two books that profoundly shaped his worldview: Aleksandr Solzhenitsyn’s Gulag Archipelago, which pulled back the Iron Curtain to expose the brutal, dehumanizing Soviet state, and J. R. R. Tolkien’s Lord of the Rings, a parable on the perfidy of power.

  At Stanford he majored in philosophy and founded the Stanford Review, a libertarian-leaning newspaper that he transformed into a campus institution that attracted national attention. Thiel also displayed enviable talents in math and finance: he could calculate square roots to the decimal point in his head. He graduated from Stanford Law School in 1992 and, after a spell with Sullivan & Cromwell, moved to CS First Boston to trade derivatives (he was hired after getting every question right on a math test). Somehow he found time to coauthor The Diversity Myth
, a book attacking the liberalization of Stanford’s curriculum. It made Thiel a darling of the conservative movement, and when he returned to California to start his own hedge fund, he did it with right-wing backing.

  The lecture that Thiel offered that day centered on the dangers of concentrated governmental power. No one despises totalitarianism more than someone who fled it, and Levchin was so moved that he worked up the courage to introduce himself. The first thing he noticed about Thiel was his blue-eyed intensity, a word that had often been used to describe Levchin. After chatting a few minutes, Thiel suggested breakfast. They met the following week at Hobee’s, where Levchin pitched a couple of ideas for start-ups, including one that revolved around cryptography. In college, he had mastered the art of creating software for handhelds and the ability to store heavily encrypted data without any loss of performance—a major feat, since these early PDAs had severely limited power, memory, and storage. Levchin proposed a company that would create a library of encryption schemes, which they could license. He figured it was inevitable that everyone would have a Palm-like device and they could capitalize on this emerging technology. Sell the encryption libraries, let others create applications; collect a penny or two per copy and let the money roll in.