Dark Mirror

by Barton Gellman

  In April, a month or so after reaching Hawaii, Snowden took on a new project. He called it Heartbeat. Snowden coded it from scratch in plain sight of his colleagues. Anyone at Kunia could follow his progress on an intranet page that listed his name and system identifier, ejsnowd, as the point of contact. At the top of the page, titled “The NSA Heartbeat,” Snowden placed a logo of his own design. Green spiky lines crossed a horizontal grid, hospital monitor–style. An overlay displayed the Kunia coat of arms, a hodgepodge of cryptologic symbols: quill pen, brass key, lightning bolt, flaming torch. The motto at the bottom read “Silent Sentinels.”

  Few assignments could have offered better cover for Snowden’s own silent mission.

  He had legitimate reason now to automate the transfer of thousands, then hundreds of thousands of files, and then more. That is far from saying he took away copies of all those files for himself. U.S. government officials, who promoted the public allegation that he stole 1.7 million documents, eventually acknowledged that their claims were based on a worst-case surmise. From personal knowledge, I can state with confidence that Snowden did not give any journalist, or all journalists combined, one-tenth that number. There are probably things he took with him that he decided not to release. Whatever the numbers, Heartbeat certainly brought a great volume of new material within his span of control. Because the project operated in open view, he said, a man assembling evidence for journalists would not “have blown up any logs, covered any tracks, hidden his trail, etc., related to the Heartbeat, as he might have if he were doing something for the public . . . and didn’t want it found.”

  Heartbeat is central to understanding the way Snowden explored and conquered his digital environment at the NSA. He discussed the project with me sparingly, and only because he believed the NSA had scapegoated his manager for authorizing it. When we spoke of Heartbeat’s workings and origins, Snowden deflected efforts to tease out detail.

  Me: help me get this right. whatever I describe, if I make the tiniest error, someone will say the story is flat wrong.

  Snowden: you don’t have to write it like a deposition.

  Me: Pot, meet kettle. Kettle, pot.

  Snowden: ha.

  just tell me the things you feel like you’re being horribly, unduly deprived of.

  I did. He was unmoved. For much of the Heartbeat narrative I rely on other people and records, public and otherwise.

  Snowden’s boss, a career civil servant, oversaw a fleet of well over two thousand Windows computers in the Tunnel. An idea “had been kicking around on the Windows team for a while,” Snowden said, but nobody had time to take it on. It went like this. Lots of people at Kunia needed regular access to information stored in faraway places, not all of them in the NSA’s digital realm. Within their assigned roles and specialties, Kunia workers might draw upon records maintained by the CIA, the FBI, the State Department’s Bureau of Intelligence and Research, or any of the other thirteen branches of the U.S. intelligence community. It could take all day to open special connections, log in, and search even a fraction of those. Some analysts had to do so often. Snowden said the overall effect was “a rat’s nest of incompatible closed networks and half-baked workarounds,” a description backed by others with firsthand knowledge.

  There was a lot of pent-up demand for a better way. Surely someone could build a one-stop portal for intelligence that spanned multiple sources? It was a simple idea, excruciatingly hard to carry out. The networks crossed the turf lines of rival agencies. They used different software, data formats, and access protocols. Each had its own intricate set of security controls, and Heartbeat would have to reproduce them exactly. If the portal worked correctly, identical searches would present each user with a different set of results, depending on her clearances and authorized need to know. This was an enormous undertaking. Kunia had no budget for it. Snowden’s employer had no contract to perform the work. “This was a self-generated idea,” Richard Ledgett, the former deputy director, told me. “It was not something ‘Big NSA’ thought was needed, so his local managers had some latitude. ‘Sure, that sounds like a good idea.’”

  The usual dodge in such a case was to put together a prototype. Maybe next year the project would win approval and funding. Snowden’s boss allowed him to give it a try. He had time on his hands and a head start after his previous work on EPICSHELTER, the backup system. Did he volunteer for Heartbeat, understanding its hidden potential? Was it his supervisor’s idea? Did Snowden subtly position himself for a tap on the shoulder? There may be no official record on that. The NSA was paying Dell, and Dell was paying Snowden, to do a different job. In reality, Heartbeat began to swallow the bulk of his time. “It’s not an exaggeration to say 70% of [my] working hours were spent on this,” he told me. One of Snowden’s coworkers, who was interviewed by Forbes long after all the crockery had broken, asked, “If you had a guy who could do things nobody else could, and the only problem was that his badge was green instead of blue, what would you do?”

  Heartbeat stretched past the boundaries of the NSA’s own systems. Like the open internet, the intelligence world’s classified networks connected a cul-de-sac in Hawaii to virtual roads and highways that spanned the globe. The fiber optic cables of NSANet, the pipes that Ledgett alluded to, linked the agency’s email servers, collection systems, processing tools, intelligence reporting platforms, and repositories of intercepted records and content. But the pipes did not stop there. Since the September 11, 2001, attacks on New York and Washington, the White House and Congress had pounded on intelligence agencies to quit hoarding information and work together. NSANet, accordingly, had hooks into an even larger system, in essence a network of networks. The Joint Worldwide Intelligence Communications System, or JWICS, bridged the Defense Intelligence Agency, the National Reconnaissance Office, the National Geospatial-Intelligence Agency, and others under Defense Department control. JWICS also linked to TS/SCI assets at the FBI and the CIA’s Agency Data Network. Everything connected to everything, even if each agency kept its most sensitive information offline. In Heartbeat’s most expansive conception, it would become a hub with spokes that extended throughout the intelligence community.

  It did not begin that way, and neither did it reach so grand a scale. Heartbeat expanded in stages, responding to what Snowden called “mission creep.” Some classified intelligence products, such as the CIA’s Intellipedia, published an automated notice each time they added or updated an article. “Originally we just mirrored those feeds,” Snowden said, “but that didn’t solve the ‘content retrieval’ problem.” If you wanted to read an article on the list, you still had to log off Heartbeat, open a network bridge to another agency, and log in to the server where the document was stored. That defeated the purpose of a one-stop portal. Snowden added a daemon, a computer process that ran in the background, to copy each newly listed document into local storage at Kunia. Then some of Heartbeat’s early evaluators asked whether Snowden could retrieve new files on remote systems that did not list them automatically.

  “Two months of paralysis” set in, Snowden said, as Hawaii’s Information Technology Directorate grappled with the implications of that proposal. The idea on the table was to create “an always-up-to-date mirror of all the latest content from all the different internal sites and networks.” In order to accomplish that, Heartbeat would have to create and update its own index of systems that belonged to other agencies. Each time the index showed something new, Heartbeat would import a copy. This would not be easy, but it was not a novel challenge in networked computing. Self-updating indexes are commonplace in the civilian world. The tools to build them are called spiders because they crawl around digital networks to look for new files. Google uses a spider of its own design to catalog the entire internet, or most of it. The other half of Heartbeat’s mission, downloading and synchronizing the new files, would adapt well-known tools such as wget and rsync.

  It was an audacious idea
to do anything of this sort on somebody else’s TS/SCI network. Among other impediments, the plan required credentials that Snowden did not possess. Every day—or every hour, or many times an hour—Heartbeat knocked on a long line of doors. Each door led to some faraway classified database. Heartbeat could not enter without an invitation, which came in the form of a PKI digital identity certificate. As a prototype with no official status, Heartbeat was not eligible for its own PKI. The system itself could not be added to the guest list. Instead, Snowden embedded his certificate into Heartbeat’s Digital Identity Store. When Heartbeat knocked on a door, it announced itself as ejsnowd. Some doors would not open for Snowden. Some of them led to places that only a government employee could go. A new question therefore arose. Would Snowden’s boss add his digital identity to Heartbeat alongside Snowden’s? Between them, the two identities would open more doors than either one alone. The supervisor agreed, Snowden said, after consulting “multiple levels of NSA and corporate management,” including the information systems security manager for all of Hawaii. Nobody objected, by Snowden’s account, but nobody put permission in writing. Projects built with a nod and a wink do not rate formal exceptions to security policy.

  Someone had to take the fall when the FBI learned that Heartbeat operated off the books. On June 18, 2013, two weeks after the Washington Post and the Guardian began to publish the NSA disclosures, investigators found Snowden’s former manager, who had left Hawaii for a new assignment. According to an NSA letter to Congress, which referred to the man only as a “civilian employee,” the manager “admitted to FBI Special Agents that he allowed Mr. Snowden to use” his digital identity “to access classified information on NSANet; access that he knew had been denied to Mr. Snowden.” That was a harsh way to spin it, stripped of context. “They threw this guy under the bus,” Snowden said. Ledgett said of Snowden’s boss, “We ended up firing the guy. He knew he screwed up.”

  In his unclassified letter to Congress, the NSA’s director of legislative affairs, Ethan Bauman, gave an account of how the transaction worked. “At Mr. Snowden’s request,” Bauman wrote, “the civilian entered his PKI password at Mr. Snowden’s computer terminal. Unbeknownst to the civilian, Mr. Snowden was able to capture the password, allowing him even greater access to classified information. The civilian was not aware that Mr. Snowden intended to unlawfully disclose classified information. However, by sharing his PKI certificate, he failed to comply with security obligations.” In plain English, Bauman accused Snowden of stealing the password by tricking a superior who should have known better. The trouble with that account is that it cannot be true. That is not the way certificates work in a system like Heartbeat. If Bauman did not know that, his agency employed plenty of people who did.

  In order for Heartbeat to function at all, it had to be connected around the clock to each remote network it watched. Connections required an identity certificate. Ordinarily, those certificates were password-protected, but autonomous systems like Heartbeat cannot use them that way. Heartbeat canvassed and copied new files in near real time, twenty-four hours a day. No one could sit by the keyboard and type a password for each of these countless events. Snowden and his manager solved that problem the way network administrators usually do for scripts and other automatic operations. They stripped the password protection from the manager’s certificate before embedding it in the Heartbeat Digital Identity Store. This was not an exotic procedure. Snowden’s boss, a seasoned network engineer and top-tier system administrator himself, could not have failed to understand what he was doing. The command may look opaque to a layman, but for these men it required no more thought than clipping a house key onto a lanyard.

  openssl pkcs12 --in bosskey.p12 --out bosskey.pem --nodes

  It translates easily enough. Launch the openssl utility. Call the pkcs12 command, which manages identity certificates. Convert the certificate called bosskey from its original format to a new one. Strip out the password with the “--nodes” option. The procedure did not steal or capture the manager’s password. It saved the certificate in passwordless form. Snowden told me he “never, at any point, knew or used the guy’s password.” Heartbeat did not need it.

  The portal grew slowly. “At the very beginning it was tiny,” Snowden said. “It had almost nothing. I had to stitch together a gigantic, massive network of all these servers and resources in order for it to have any value, before it was collating any kind of information at all. . . . I don’t think that would’ve been happening in a meaningful way until probably 2013.”

  * * *

  —

  On January 24, 2013, Snowden read an alarming post in a classified blog devoted to fresh advances in NSA surveillance. Over in the agency’s Application Vulnerabilities Branch, S32313, a small group of clever geeks had come up with a way, in some circumstances, to break the anonymity provided by Tor. That was the privacy network that Snowden helped support, the one he taught to novitiates at the cryptoparty in Honolulu. More to the point, he was staking his freedom on Tor as he contacted journalists. His conversations with Laura Poitras had begun three weeks before, and Poitras had already come to see me in New York. If Tor could be compromised, so might we all.

  For years, the NSA and the GCHQ, its British counterpart, had been banging on Tor, looking for some way to pierce its veil of anonymity. It was a hard target, among the most intractable of the tools available to the public at large. (“Tor sucks,” one NSA presentation slide complained.) In order to make it easier for novices to use, Tor developers had baked their magic into a custom version of Firefox. They called it the Tor Browser Bundle. Now a small team of NSA hackers had come up with a way to see through the browser’s privacy shields.

  The NSA liked to hire rising stars in computer science and mathematics as interns for a summer or an academic year. Young innovators came up with hacks that old-timers missed, and a taste of life inside sometimes hooked them into postgraduate employment. It was one of those interns who broke the news that Snowden read in January. The intern and his team had found a vulnerability in Firefox, not in Tor itself, but they could exploit it against certain versions of the bundle. It was a mark of Snowden’s icy nerves—and the betrayal that some colleagues felt so strongly—that he congratulated the intern and began a correspondence to tease out details.

  “I read your journal.nsa entry,” Snowden wrote to the intern. “Really great work! This looks like an awesome way to deal with the TBB. I’d like to know more if you don’t mind a couple quick questions.” Did the method rely on prior identification of a target, or could it compromise any Tor browser? Did it work against all operating systems? Were there any browser plug-ins that blocked the exploit?

  The intern happily entertained the shoptalk. “Thanks!” he replied. “I’ve attached my current draft of slides.” The densely technical, fifty-eight-page presentation laid out the methods and limits of the new exploit, which had the cover name EGOTISTICALGIRAFFE, or EGGI for short. At the moment, the intern told Snowden, NSA operators were deploying it “only against certain extremist web forums,” but he added, “I am under the impression that they can serve up an exploit to pretty much anyone.” EGGI worked on the Windows browser but not on Mac or Linux. It did not work at all if a user switched off JavaScript, a programming language built into modern web browsers.

  Snowden was safe. He always switched off JavaScript. The Tor browser made that easy enough, but the scripting language was left on by default. Without explaining why, Snowden pointedly reminded me and Poitras in coming months to “disable the fucking scripts.”

  Snowden kept the conversation going with the intern. He had a pedagogical point in mind, and he wanted to save it for later.

  “Seriously, this is really great stuff,” he wrote on January 25. “I hope you get all the kudos you deserve for putting this together. How long did it take you to come up with this? If Tor team updates [Firefox] . . . do you think the TBB has enough target surface for you g
uys to restore the same access through a different vuln? Same time investment, or more?” Snowden was asking how long it would take to find a new hole in the browser once the old one was patched.

  “Somewhere between a week and two weeks,” the intern replied. “We’ve actually got a couple of bugs we’re looking at for Firefox 17+, and once I get back from a TDY [temporary duty] next week, I’m going to try to work with the rest of the team to get something ready to ship, and I’m confident we can have it ready when [Tor developers] release something new, or very soon after:)”

  Much later, when government officials and other critics accused Snowden of damaging NSA collection capabilities, he cited the exchange with the intern in reply. “What I was actually doing was, I was trying to elicit information for the benefit of you guys on how long it takes,” he said. “After they’ve lost their exploit, how long does it take them to build another?” That question, he said, “is actually a really great metric against the government’s argument that we’re going dark.”

  * * *

  —

  In the first quarter of 2013, Snowden was presented with two prospects for the final job of his intelligence career. Someone suggested he take the test to join the NSA’s Tailored Access Operations unit, or TAO, a position that would shift him back from Dell to U.S. government employ. “Tailored access” meant breaking into specific networks or machines that eluded the NSA’s large-scale collection systems. Most of the tailors’ work was routine. The agency had all kinds of off-the-rack hacking ensembles. The tailors worked from checklists. If this, then that. Survey your target’s hardware, software, firmware. Identify known security holes. Check for antivirus applications. Push this button, run that scan, deliver the corresponding exploit. TAO slang for this was “popping boxes,” no harder than popping a door lock for a halfway competent burglar. Boxes were the computers, routers, and firewalls of a surveillance target. With aptitude and training, a newly enlisted cryptologic technician could do the job most of the time.