Tubes: A Journey to the Center of the Internet

by Andrew Blum

  Their networks are headed directly to Internet exchanges (not your home)—an architecture that only makes the exchanges more important. If you’re going to bother building out your own network, it needs worthwhile places to go: good regional distribution nodes. A network will connect directly into a place like Ashburn, where its owner will hang a shingle to announce its willingness to interconnect with other networks. In some cases, it’s actually a shingle: a printed placard looped on to the outside of a cage, to attract the attention of visiting network engineers. More often, it’s a virtual shingle: a listing on a website called PeeringDB, or just an information page of its own, as Facebook has.

  Facebook.com/peering isn’t behind any password, or within some proprietary database. It’s wide open—as exposed as your cousin’s vacation photos. A brief paragraph at the top describes Facebook (for the benefit, one supposes, of any peering coordinators from Mars): “Facebook is a social utility that helps people communicate more efficiently with their friends, family and coworkers.” Then it lays out the company’s peering m.o.: “We have an open peering policy welcoming the opportunity to engage in peering with any responsible BGP speaker in an effort to improve the experience of our millions of users throughout the globe.” Being a “responsible BGP speaker” means you know how to configure a big Internet router and can be counted on to fix it fast if you screw up. Its “open peering policy” makes Facebook a classic peering slut, happy to connect with all comers. Then there is a table that shows where you can connect, listing sixteen cities around the world, the particular Internet exchange in those cities, the IP address (like the Internet’s postal code), and the capacity of the “port” at that location.

  The first time I saw Facebook’s list—during a coffee break at NANOG—my eyes widened. For months I’d been talking to network engineers and facility owners, polling them about the Internet’s most important places, working up a judgment about where, exactly, one looks for the Internet. And then here, wide open to the world, was exactly that—at least according to Facebook (the world’s second-most-visited website, after Google).

  Facebook is not in the business of delivering Facebook pages to people’s homes, offices, or cell phones; it relies on other networks to do that. This page said that if you run one of those networks, and you’re “responsible,” Facebook will connect to you, either directly (router plugged into router like at the PAIX or in Ashburn) or via a central switch (at an Internet exchange). It’s Facebook’s attitude that the more connections, the merrier, which makes this public information in the same way that American Airlines will happily tell you where it flies. So if you’re a small ISP in, say, Harrisburg, Pennsylvania, and you’ve noticed that a meaningful percentage of your traffic is coming from Facebook, and you’ve built your network back to Ashburn anyway, you’d strongly consider asking if you can plug in directly. Facebook’s peering coordinator will probably say yes. And your customers will notice that Facebook loads faster than almost anything else.

  Yet what’s most telling about Facebook’s peering list is how short it is. The global capitals aren’t surprising: New York, Los Angeles, Amsterdam (at AMS-IX), Frankfurt (at DE-CIX), London (at LINX), Hong Kong, and Singapore. The big US cities—Chicago, Dallas, Miami, and San Jose—are also to be expected from an American company. But the small US cities put in stark relief the unique geography of the Internet. When else is Ashburn, Virginia, on the same list as London and Hong Kong? Or Palo Alto? Vienna, Virginia (also on the list), is next door to Tysons Corner, which (for the moment) still has enough gravity to draw a crowd. It’s clear that the geography of these buildings can partly be understood on a global scale: the Internet follows its users, meaning all of us, to where we live. But it’s also clear that if you zoom in on the map, those broader forces fall away and are instead replaced by the ad hoc decisions of a small number of network engineers, each searching for the most technically and economically efficient places to connect. Palo Alto or San Jose? Ashburn or Vienna, Virginia? The thing about peering is that its influence cuts both ways: networks want to be at places where there are lots of networks. Facebook’s choices of locations are therefore both a response to the growth of a given location, and a seed to its future growth. Or maybe it just likes the blue lights at Equinix. Or the beer in Amsterdam. Or both.

  The chatter about peering came to a head on the last full day of the NANOG meeting in Austin, at a session listed somewhat cryptically on the agenda as the “Peering Track.” Its late timing was deliberate, when the network engineers were hopped up like schoolkids on a Friday afternoon. Inside the meeting room, the banquet chairs had been arranged in a circle, ostensibly to facilitate discussion, but more, it seemed, to create a gladiatorial atmosphere.

  This was peering speed dating: Internet exchanges advertised their size and prowess, and peering coordinators advertised to one another. They called it “peering personals.” A successful pitch in front of this crowd might lead to a chat afterward, and a new route after that. Perhaps you run a data center in Texas, but you happen to have a big Danish website as your customer. Peering with a Danish ISP might get a lot of traffic off your hands—enough to make it worth meeting that ISP in Ashburn. That’s exactly what Nina Bargisen, a network engineer at TDC, the Danish telephone company, had in mind when she put forth this simple plea: “I have eyeballs eyeballs eyeballs,” she said. “For all of you with content, please send me an email.”

  Dave McGaugh, a network engineer at Amazon, did his best to dispel his colleagues’ expectations that his traffic ratios were skewed in his favor: “We are outbound heavy but not to the extent people might expect,” he pleaded. Will Lawton, the representative from Facebook, was understandably aggressive in his offer; Facebook, after all, has a lot of news to share. Facebook’s typical “ratio of egress to ingress” was 2:1, he said, assuring any skeptics that he’d be accepting plenty of their traffic (like uploaded photos) for every bit he’d be sending back (the viewed photos). Across the board, the message from the peering coordinators was “peer with me.”

  The message from the exchanges was “peer at me,” make that physical link in my city—make my place your place. The competition was intense, especially among the biggest. The audience was hushed as in back-to-back presentations the London, Amsterdam, and Frankfurt exchanges each highlighted their recent growth, offered a few words about their robust infrastructure, and finished with a pitch about the importance of their place in both the physical and logical worlds. The smaller exchanges followed, left only with the comparative advantage of their real-world geography—their ability to solve each network’s particular “Chicago problem.” As Kris Foster, the representative from TORIX, in Toronto, offered, “If you’ve got fiber routes from New York to Chicago through Toronto, maybe you should think about stopping.” It wasn’t a bad suggestion. It might make your network more efficient, but at the least it was a way to be “multi-homed,” a salve against fragmentation.

  But for the most part, the gravitational pull of the biggest exchanges was powerful enough to overcome that geographic diversity. The network effect was unassailable: more networks disproportionately concentrated in fewer places. The result was a striking gap between the average-sized Internet exchanges and the behemoths. “Places like AMS-IX and LINX are a routing engineer’s paradise,” Renesys’s Cowie explained. “They are full of hundreds of organizations that are begging you, ‘Please look at my routes, study me!’” That’s irresistible for a NANOGer. The big exchanges become bigger, and it seems likely they will stay that way, growing in proportion with one another. That made my job a little easier: the Internet’s map was, for the moment, fixed. My itinerary was clear. If I wanted to see those singular boxes at the center of the Internet, then Frankfurt, Amsterdam, and London were the places to go. What I didn’t know yet was how different they would be. Or if they accepted visitors.

  That evening Equinix sponsored a party at an Austin music club with a huge roof deck. A large “E” was projected on the danc
e floor and a woman at the door handed out guitar-shaped Equinix key chains, in honor of the city’s famous music scene. It wasn’t over-the-top extravagant, but it was the biggest party on the closing night, with enough free drinks to ensure that none of the NANOGers would miss it. For Equinix, hosting it was a no-brainer. The hundred or so network operators milling around drinking beer each represented a network, and they were more likely than not all Equinix customers—if not several times over. Even better, if two network engineers met at this party and then decided to peer, the decision would be consummated with a “cross-connect,” a cable from one cage to another—which meant recurring revenue for Equinix. The key chains were the least it could do.

  I had my own agenda. I didn’t have a network (excepting the dusty one behind the couch), but rather an image of all the networks, a giant imaginary map I had been steadily filling in. During the peering session, the AMS-IX presentation had been given by a young German engineer with a shock of red hair. Her boss, a jovial and slightly rounded Dutchman named Job Witteman, had watched quietly from the side, looking a bit like the Godfather, leaning back in his chair. He was the guy I wanted to talk to. NANOG attracts a brainy crowd with strong opinions, not always gracefully expressed. The session Q&As are almost always contentious. Shouting matches aren’t uncommon. (The self-consciously combative tone of the peering session was meant to neutralize some of that.) But Witteman gave the impression of rising above all that drama, an elder statesman with his head above the fray. He didn’t seem like an engineer.

  “I never touched a router in my life!” he yelled over the music, when I asked, at the party. “I know how to switch on a computer, that’s about it. I know what a router can do and how it functions—but don’t ask me to touch it. That’s what other people are for.” He ran one of the largest Internet exchanges, but he’d always avoided learning the nitty-gritty of networking. The strategy worked; it was one less thing to argue about—and AMS-IX was known for its technical competence. Witteman sketched its history for me. Like many exchanges, it was founded in the mid-1990s as an offshoot of its nation’s early academic computer networks. But unlike most, it was quickly professionalized. Rather than letting volunteers handle technical support, AMS-IX treated the task from the beginning with the same precision and planning as the Dutch treat everything else. “The whole purpose of us building a business back then was to become professional,” Witteman said. “We didn’t want to be like, ‘Who’s in charge today?’ ”

  From early on, he took an equally Dutch approach to creating a marketplace. As AMS-IX shook off the mantle of the computer science departments, its spirit became intensely commercial—yet communal. “It’s somewhere in the genes in the Netherlands. We are good at trading organizations, at exchanges—like tulip bulbs and whatnot. We don’t need to buy and sell, but we like to create the marketplace,” Witteman said. A very open marketplace. On the AMS-IX, as on the streets of Amsterdam, you could do whatever you want—so long as it didn’t bother anyone else. “We’ve always been the company where we say, ‘This is your platform, you pay for it, your port size determines the amount of traffic you can flow over the exchange, but we don’t look into it, we don’t care, we don’t mind.’ ” It struck me as a pure expression of the Internet’s publicness, this interconnected group of autonomous networks left to their own devices in a carefully managed setting. It also recalled the Internet’s founding ideas, vaguely Californian, of live and let live, “be conservative in what you send and liberal in what you accept.” To a certain extent this was true not only of AMS-IX but all Internet exchanges.

  It was an openness that came with risks. There’s little doubt the scourges of the Internet—child pornography chief among them—cross the AMS-IX, and others. But Witteman was adamant that it wasn’t his business to prevent it, no more than the post office was responsible for what it carries. When the Dutch police once asked to plug in tapping equipment, Witteman carefully explained that it wasn’t possible—but they could become a member of the exchange if they wanted and peer individually with the ISPs they’re responsible for policing. “Now they’re paying for their port and everybody is happy,” Witteman said, taking a swig and raising his eyebrows.

  As we talked, a tall man with a trim beard and spiky hair joined us, thumbing at his smartphone purposefully while he waited for a break in the conversation. When it came, he turned the little screen toward Witteman and shook his head with feigned surprise. “Eight hundred,” he said. Witteman responded with wide eyes, as if with mixed emotions. The tall man was Frank Orlowski, Witteman’s counterpart at DE-CIX in Frankfurt. He meant eight hundred gigabits per second, his exchange’s peak traffic that afternoon, another new record. More than Amsterdam. Ten times more than Toronto.

  Clearly the competition among these big exchanges was at least a little personal. Orlowski and Witteman were on the circuit together—sometimes co-conspirators, oftentimes friendly competitors. They’d both made the journey across the Atlantic to Texas, having crossed paths just a few weeks before at a similar event in Europe. Witteman was undoubtedly jealous of DE-CIX’s growth, but he was equally proud and amazed that their baby—by which I mean the Internet itself and the exchanges that sit at its center—had grown so big. As I listened to them, I loved how intimate it made the Internet’s infrastructure seem, how the queries and messages of an entire hemisphere could be understood in the clink of beer bottles in a bar in Texas. The social binds that tied the physical networks together were visible, not just among Witteman and Orlowski but across the whole crowd at the party. I can’t say I was surprised that the Internet was run by wizards—it had to be run by somebody. But I was surprised by how few they were.

  But what about the places, the physical components, the hard ground? Was the Internet as concentrated as it had seemed during the peering session, or looking around the bar that night? At NANOG, the engineers were focused on just a handful of cities, with a clear hierarchy. Was this the Internet’s geography? Frankfurt and Amsterdam were as far apart as Boston and New York (which isn’t very far); Witteman and Orlowski ran equally professional operations. They both put in the effort to rub shoulders here, thousands of miles from home, encouraging engineers to peer at their exchanges. What would make a network engineer choose one over the other? Would I be able to tell the difference between the two places?

  These big exchanges seemed a distillation of the Internet’s essence: single points of connection that sparked new connections, like a hurricane gaining power over the ocean. I wanted to be reassured that even though the Internet made places less relevant, its own places still mattered—and with it, perhaps, the whole corporeal world around me. I’d come to NANOG to meet the people who individually ran the networks and collectively ran the Internet. But what I really wanted was to see the places where they meet, to somehow get closer to understanding the physical geography of my virtual life. Witteman and Orlowski’s pieces of the Internet were inherently rooted in their own places—as distinct as their own national identities. Where did all those wires lead? What was there actually to see?

  I told Witteman and Orlowski about my journey, before raising what I hoped was the obvious question: Could I come see for myself? “We have no secrets,” Witteman said, cartoonishly looking inside his jacket flaps to emphasize the point. “Anytime you want to come to Amsterdam.”

  Orlowski looked down at us both and nodded in agreement. I was welcome in Frankfurt as well. We toasted with our Equinix beers.

  On the late winter day I arrived in Germany, the gray sky perfectly matched the steely bank towers that rose up beside the river Main. I spent a jet-lagged Sunday afternoon exploring Frankfurt’s quiet center. In the cathedral, I saw the side chapel where the kings of the Holy Roman Empire would gather to choose an emperor. From that single room, the news was sent out across the land. Nearby was a more contemporary landmark: the large statue of a blue-and-yellow euro symbol, famous as the backdrop for news reports about the European Central Bank. Both chapel and
euro hinted at the particular spirit of this place: Frankfurt has always been a market town and communication hub, a sternly self-important city.

  That evening I had dinner at a hundred-year-old restaurant with a friend, an architect and Frankfurter. Over beef with traditional green sauce washed down with pilsner, I pressed him to help me understand the city, and how its big piece of Internet fit in with the whole. But he mostly demurred. Frankfurt is not a place given to romantic aphorisms. It is short on homegrown anthems and atmospheric poems. It does not often appear in films. Among its most famous sons are the family Rothschild, the great German-Jewish banking dynasty (whose success, appropriately enough, came from strategically fanning out around the world—and using carrier pigeons for speedy long-distance communication), and Goethe (but he hated the place). Among Frankfurt’s greatest contributions to twentieth-century culture is the “Frankfurt kitchen,” a kitchen design of supremely utilitarian character, even by the standards of the Bauhaus (it sits in the collection of the Museum of Modern Art). If anything, Frankfurt is best known as a place for trade fairs (a role it’s played since the twelfth century), like the Buchmesse for books and the Automobil-Ausstellung for cars, and for its big airport, among Europe’s largest hubs. So I wasn’t entirely surprised when my friend eventually came to a simple and resonant observation: Frankfurt was a transient city, a place where people did their business and then left. Despite its five million inhabitants, Frankfurt was not a place to truly live. And that turned out to be true for the bits as well.

  The next morning I went to DE-CIX’s offices in a brand-new building of glass and black steel, overlooking the Main in a stylish neighborhood of design stores and media companies, nearer to the shipping docks than the banks. The DE-CIX network engineers worked in an open room in the back, their desks set among whiteboards on wheels, scattered purposefully like the trucks on an airport tarmac. But this was only DE-CIX’s administrative home. The “core switch”—the beating heart of the exchange, the big black box through which those 800 gigabits per second of traffic flows—was a couple of miles down the road, and its backup was equidistant in the other direction. We’d pay our respects after lunch.