Hacks

by Donna Brazile

  I felt like the Hacker House never turned their lights off. I was always one of the last people to leave at night, and I always saw the glow of lights in the room called “the Thunderdome.” I worried that we were exploiting them. They were putting in impossible hours. I felt bad, but they said my concern was misplaced. That’s the kind of job we love, they told me. To find all the things that are going on that are horrible, while remaining as optimistic as possible.

  I could not believe how lucky we were to have these patriots volunteering their services to rescue us in the crucial last weeks before the election. I suppose politics has one similarity to the spirit of these cyberwarriors. Our job is to face the things that are horrible and provide hope. Yet we had come to a feeling of hopelessness and constant struggle against an elusive foe. I was not talking about the GOP candidate. I meant the Russians. After a week of these twice-daily phone calls, I felt confident that this part of our crisis was in the hands of experts.

  Even the Hacker House team sensed the low mood of the DNC and wanted to do what they could to boost it. They wanted the staff to feel safer in their daily lives. The Hackers decided to become a more visible part of the DNC team. After one of the staff dropped by the computer room to tell Andrew that someone was trying to extort him through his computer, Rob worked with his girlfriend, a graphic designer, and his friend Samantha Davison, a security outreach expert, to develop a flyer. It encouraged the staff to take precautions and to come to them if anything in their computer life went awry. They placed this flyer over every toilet and urinal in the DNC. You couldn’t miss it.

  I thought I was as aware as the average person about how to keep my devices safe. I had learned a tremendous amount about security since my world had been torn apart by the hacking. These recommendations taught me even more:

  STAY UP TO DATE—Malware depends on you working on outdated apps and devices, so staying current helps you prevent being a victim of the hacking.

  2-FACTOR AUTHENTICATION—Make sure that every time you sign on to any of your networks or social media you verify that through a text message or email message to your cell phone.

  SECURE YOUR PHONE—install LOOKOUT, an app that scans your devices for malware regularly.

  SHARE THROUGH THE CLOUD—Email is not safe and thumb drives are easily corrupted. Cloud services have security teams that work round the clock to protect your data and are the best way to keep it safe.

  Even if a flyer might sound silly to you, this was a huge boost to the staff. People suddenly had a plan, and with a plan they felt like they could do something to handle their anxiety about the hacking. Staff started to stop by the Thunderdome for advice and the Hacker House crew became a source of optimism for all of us.

  After they cordoned off the The MIS Department system from ours, the team’s next effort was what they called tabletop exercises. Since the Hacker House team knew they were going home at the end of the month, they needed to teach the in-house technology staff how to analyze the telemetry so that they could respond to an attack. With the new visibility created by Tom’s logs, the team staged tabletop exercises, like cyber–fire drills. Hacker House would devise an intrusion and announce it at the tabletop exercise so they could observe how the staff responded to the event.

  The first tabletop exercise, during the second week they were in DC, was a disaster. The staff even had trouble signing on to the system because of the new ways it had been configured by Hacker House. The steps the staff needed to take to investigate the breach of the system were not intuitive to our IT staff. The team had to coach them almost every step of the way, asking them leading questions and giving them broad hints.

  There was also an emotional issue underneath the incompetence, Heather recognized. There was some mistrust among the staff. People had made mistakes. They felt bad about those and about themselves, but they also had been traumatized by the results of those errors, as had the people they worked with. Heather sensed that the tech staff was holding back, not being as forthcoming as they might be, because they didn’t want to be blamed further for the situation we were in. It took many tabletop exercises and several weeks before that feeling melted away.

  As the Hacker House team grew from two to five, when they discovered other problems that they needed to handle quickly, they reached out to top specialists in other areas. Some specialists they recruited were former DNC staffers who knew how to rebuild parts of the system while it was still running. Rob boasted that the team had a 100 percent recruitment rate. These top security engineers are much in demand, the hardest to hire in the industry, and usually stay at a firm for only two years before they get bored and want another challenge. Every single person they asked immediately said yes, dropped what they were doing, and flew to DC. By the end of the month sixteen engineers helped the Hacker House effort, some staying for a few nights or a full week, some sleeping on the floor of the old row house we rented for them in Northwest DC, while the core group of five remained virtually the entire time.

  The Hacker House developed its own culture. They all suffered together when the Soylent bars were recalled by the manufacturer. Rob and Ryan were working so hard that they didn’t hear about the recall, and the bars made them sick. Evenings when they got off before ten they sometimes went to Ted’s Bulletin, because they liked the old DC atmosphere and the milkshakes and Pop-Tarts. Maybe a little too much. One night when they were home at the row house, they tried to order from Ted’s Bulletin, but the restaurant was out of their favorite flavors and kept calling back to cancel different parts of the order. They reached out to other places to get their milkshake-and-Pop-Tart fix. By the end of the night, amid all the confusion, they ended up with $100 worth of milkshakes and Pop-Tarts from two different places.

  Each night when they got back to Hacker House, besides the video games they played, the crew would often open up their laptops on the dining room table and share with each other the techniques and skills they had learned over the years. “It was an egalitarian atmosphere of people operating at the highest level,” said Chris Long, who joined the house in mid-October. “I learned more in ten days there than I did in two years at work.”

  Right after ace hunters Chris and Ron arrived, we had our second big incident. On October 20, Falcon picked up an intruder using credentials that were only entrusted to system administrators and the team feared that this intruder, whom they named Airwolf, the name of the system that had been compromised, was attempting to exfiltrate data from the cloud. They spent a full week tracing back how those credentials had gotten into the hands of someone outside the system.

  During this time, even Heather got a little spooked by the work they were doing. The problem of credentials was supposed to be fixed before the team arrived, but it hadn’t been. One of the HH team compared it to not changing the locks on the front door after the house has been robbed. The team began to fear that there were many backdoors into the system and that they could not trust any of the remediation that had been done by the DNC.

  Heather said she never thinks about who the adversary is. She likes to focus on the intellectual stimulation of picking the problem apart, as did the rest of the team. To them it did not matter if the intruders were Russian or someone else, and besides, figuring out who was the intruder was not their specialty. They believed the FBI experts, but never verified whether the hackers were Russians or not. During the Airwolf crisis, she joined our now daily call while she was in London. It was ten in the evening there, and she had dodged outside to a chilly alleyway to get some privacy. It suddenly hit her, as she stood in this public place with people passing by on the street around her, that she was discussing the fate of the U.S. election. Heather recognized she had to be very careful about making sure our conversations were encrypted. “It was the first time in the work I’ve done that I felt personally unsafe,” she later told me.

  And then on October 21, they discovered Raider.

  Chris discovered malware on Raider, the most important se
rver in the whole system. Raider was the server that all the other servers backed up their data through. Any malicious entity that gained access to Raider essentially had the keys to our whole digital kingdom. When Chris discovered malware still running on it, the team was shocked. They thought Raider had been taken off the network when the DNC remediated the hacking, but there it was still trying to make connections to servers in a foreign country.

  With the discovery of malware on Raider, the team realized the scope of this attack might be much larger than predicted, placing the core of the DNC’s systems at risk. Heather flew to DC and worked alongside the Hacker House crew for the first time.

  One of the things that hackers needed to do was a forensic investigation. Each piece of the network produces data for every action that it takes, and those actions remain on logs that are kept within the system. Those who specialize in forensics know how to analyze these logs for suspicious patterns, like a piece of software that sends out a signal every five minutes or every half hour. Legitimate pieces of software might do this, too, but the malware does so to connect with a server owned by the intruder. In order to see these things, the computer logs have to be sorted and characterized so that the analyst can recognize the signal from the noise. Chris said when he looked at the possibility of sorting out this data from Raider’s disc, he thought it would probably take him two or three days. Heather offered to do it for him. She was so skilled at this task she pulled it off in a few hours.

  After analyzing the telemetry they understood that, if what they feared was true, they needed to act defensively. The intruders had been sitting in our voter data files for months. They had downloaded a lot of information, but they also could have manipulated what was there. Two weeks from the election, Heather and Ryan came into Tom McMahon’s office to request permission to take the whole system off-line so that they could do the forensics necessary to determine if the hackers had exploited the vulnerability in Raider to manipulate the DNC’s voter data. Tom was alarmed, fearing that in these crucial last days before the election the DNC system was giving campaigns false data.

  “What does this malware do? Are we campaigning in the wrong places? Sending flyers to the wrong houses? Are we calling the wrong people on the phone? Are we sending Hillary to the wrong states? How do you know what this malware is doing?” he asked.

  Heather and Ryan said they could not be sure, which was why they wanted to take the system off-line for four or five days to investigate. Tom agonized about this. All of the Democratic Party used these files. Were we to take the system off-line for any period of time this would cripple our election operations nationwide. Soon thereafter the whole country would know about the problems inside the DNC.

  Tom talked it over with Heather and Ryan and as he did he started to calm down. If they had been calling the wrong houses, the party would know that right away. He decided that we could not take the system off-line and would have to work with the situation as it stood.

  As the team examined Raider further they discovered something frustrating about this malware. When you download a piece of software onto your computer system it resides on the hard drive. When you run the software, it makes an exact copy on the computer’s memory. What Chris discovered was that the original version of the software was nowhere to be found on the hard drive. The malware that they discovered existed only in the memory of the computer. Because it no longer existed in the file system, extracting it would be extremely difficult.

  Well, so what? I thought. Just get the damn thing out of there and quick. Not so fast, the Hacker House team said. Raider was so old that the existing tools to extract it from memory were not guaranteed to work. And, if we did not do this carefully, there was a good chance that extracting it would crash the system and erase the memory, including the only remaining copy of the malware. If that happened the Hacker House team would not be able to pull this piece of malware out to analyze what damage it had caused. Nor would they be able to send it to the FBI, as they did with all malware they discovered, to aid in the bureau’s investigation of the hacking. Plus, they had to work quietly, because if the intruder detected that Hacker House knew about this piece of malware, they were likely to shut it down and switch to using other tactics and techniques that would be harder to detect. What they had to do was the cyberequivalent of brain surgery on an awake patient.

  This was an enormous problem that needed a swift response, but it had to be done with great care. All the Hackers reached out to their colleagues in the cyberworld to get advice. The CrowdStrike team came to Thunderdome so that they all could work together to game out a response. After careful planning and consultation, plus several simulations, the extraction took only a few seconds. It worked.

  At the same time that the team was wrestling with the Raider problem, they continued to train the staff through tabletops. As the staffers’ skills improved, so did their confidence and their team spirit. Hacker House had to get more sophisticated in the challenges they created for the staff.

  During those late nights at Hacker House, the team planted benign malware in the system for the staff to find during the tabletop exercises. At one point, CrowdStrike found the malware that Hacker House planted for the staff to discover and called them on it, as if this was a prank. Crowdstrike wanted Hacker House to stop these stunts because they were stressing out Crowdstrike employees during a critical period. They compromised when Hacker House agreed to warn Crowdsrike before they planted malware so that Falcon would not be caught off guard.

  As October came to a close, the staff needed less and less guidance from the Hacker team. The mood at Thunderdome was upbeat and so were our daily calls to review their progress, sometimes with Derek Parham, the deputy CTO of Hillary’s campaign, also joining the call. The staff had learned so many advanced skills from Hacker House and were incredibly grateful to them. Hacker House was proud of the distance the team had traveled in such a short time, as well as being grateful to the IT staff for their energy and support. And when Raider was finally out of the system, the team seemed practiced enough to respond to incidents on their own. The hackers knew at last they could go home, telling the staff at Thunderdome to remember that they were only a phone call away.

  We had a celebratory dinner on October 26 at Del Frisco, a DC steak house, the night before the Hacker House team departed, and drank a lot of very fine Japanese whiskey. I could not adequately express our gratitude to our rescuers. It had felt as though we had no one to help us until Hacker House showed up. They were one of the most remarkable things to happen in my life in politics, and until now they have received no public credit for their work.

  The DNC vs. the Russians was never a fair fight, but these people made us come out the victors. This was not the kind of triumph where you see the perpetrators yanked away in handcuffs, and we knew that they would be back again. They left us with the tools and the methodology to face the next intrusion and with a big jolt of confidence that we stood a fighting chance.

  There was no way to pay the Hacker House team for what they had done, but I would never forget their service.

  EIGHTEEN

  Comey’s 18-Wheeler

  Early on Sunday morning two weeks before the election, the technician I hired to sweep the DNC offices for listening devices arrived with his equipment. We’d had the place swept in September, but a lot had happened since then: the rise in personal attacks on me, the fact that Hacker House and CrowdStrike had discovered an uptick in outsiders trying to scam the staff’s email addresses, and the mole sent to harass us by James O’Keefe. Of all the things I was worried about at that time, the security of the staff was never very far from my mind.

  Every day I thought about Seth Rich. I had his picture on the wall of my office, along with our poster offering money to anyone who would help us find his killers. People in the office mentioned him to me frequently, some because they still missed him terribly and others because his death had made them feel unsafe. In the precious few days befor
e the election, I did not want to increase the staff’s anxiety about their personal safety. This was why I had the technician come very early on a Sunday, before anyone would be at their desks. He could get in and out before anyone besides me and a handful of the senior staff knew what he was up to.

  An odd incident had occurred the week before the security sweep. The senior staff and I were meeting in Debbie’s office when a woman I’d never seen before walked onto the patio area adjacent to the windows and started watering the plants, and then moved from there to start watering the orchids in Debbie’s office. I did not know whether we had a contract with a gardening service, so I asked the staff. No one else knew, either. I asked Charles and his staff to search the records for one. They tried for days, but they never found one.

  I didn’t wait for that answer. I started moving those plants away from the chair’s office. I put some in the reception area, and some more near the water fountain. I was concerned about bugs, and not the kind where you need pesticides. That was when I decided to have the place swept again.

  The technician had been trained by the Department of Defense, NSA, and the Secret Service to perform this job. I only gave him a two-hour window, and he said he hoped that would be enough. It was important to me that he take extra time in the executive offices, particularly around Debbie’s office and mine. He moved swiftly through the building with his ultrasensitive microphones, scanning for any anomalies in the radio frequencies. His scan was routine, uncovering nothing, until he got to the patio outside Debbie’s office and mine, the place where the interns took their breaks and ate their lunches.