But the final payload those saboteurs had planted, to Yasinsky, was a kind of black box. He could see that the hackers had, ahead of their midnight strike, installed a collection of dynamic-link library, or .dll files, essentially collections of instructions they could call upon. But industrial control systems are their own arcane discipline within cybersecurity, and Yasinsky, despite his knowledge of the forensics of traditional IT systems, couldn’t interpret the .dll files himself. Krotofil, his friend and go-to industrial control systems expert, had helped to guide him through that side of the Ukrenergo investigation. But thanks to the nondisclosure agreement he’d signed with the utility, he couldn’t share the .dlls with her.
Yasinsky showed the files to Ukrenergo’s engineers, and they told him that the code included commands written in a particular protocol—a kind of computer vocabulary understood by their circuit breaker equipment. Somehow, those files had triggered the final, disruptive step of the hackers’ blackout operation. Exactly how would remain a mystery for months to come.
* * *
■
In the United States, meanwhile, the second Ukrainian blackout resonated momentarily through the cybersecurity community, stealing back a modicum of attention from the frenzy around Russia’s election-focused attacks. For the first time in history, as Lee described it to me, a group of hackers had shown it was willing and able to repeatedly attack critical infrastructure. They’d refined their techniques over multiple, evolving assaults. And they’d planted their malware on the U.S. grid once before.
All of that meant, Lee argued, that American utilities and government officials needed to see Russia’s escalating cyberwar operations not only as Ukraine’s problem but as their own. “The people who understand the U.S. power grid know that it can happen here,” he told me.
When I’d run that notion by NERC’s chief security officer, Marcus Sachs, in a phone call, he’d downplayed the threat. American power companies have already learned from Ukraine’s victimization, he argued. Sachs pointed to the road show of briefings he and others had performed for U.S. utilities to educate them about the attacks, hammering into them that they need to shore up their basic cybersecurity practices and turn off remote access to their critical systems whenever possible. And for all the sophistication of the Ukraine grid hacks, he pointed out, even they didn’t really constitute a catastrophe; the lights did, after all, come back on.
“It would be hard to say we’re not vulnerable. Anything connected to something else is vulnerable,” Sachs said. “To make the leap and suggest that the grid is milliseconds away from collapse is irresponsible.”
But to hackers like Sandworm, Lee countered, the United States could present an even more convenient set of targets. U.S. power firms are more attuned to cybersecurity, but they’re also more automated and modern than those in Ukraine, with more computer-controlled equipment. In other words, they present more of a digital “attack surface” to hackers than some older systems.
American engineers, he argued, also have less experience with manual recovery from frequent blackouts than a country like Ukraine. Regional utilities in Ukraine, and even Ukrenergo in Kiev, are all far more accustomed to blackouts from the usual equipment failures than American utilities. They have fleets of trucks ready to drive out to substations and manually switch the power back on, as Ukrainian utilities did in 2015 when the hackers first hit them. Not every hyper-automated American utility is prepared for that all-hands, on-the-ground manual override. “Taking down the American grid would be harder than Ukraine,” Lee said. “Keeping it down might be easier.”
As Sandworm’s power and brashness grew, the question remained: Would it ever dare hit the United States the way it had Ukraine? An attack on American utilities, after all, would almost certainly result in immediate, serious retaliation from the U.S. government, even if the same attacks in a regional war of Russian aggression had barely elicited a murmur from U.S. officials.
Some cybersecurity analysts at the time of Sandworm’s second grid attack argued that Russia’s goal was simply to hem in America’s own cyberwar strategy: By turning the lights out in Kiev—and by showing that it’s capable of penetrating the American grid—Moscow had sent a message warning the United States not to try a Stuxnet-style attack on Russia or its allies, such as the Syrian dictator, Bashar al-Assad, whose revolutionary opponents the United States was supporting in the Syrian civil war.
In that view, it was all a game of deterrence. As one influential pseudonymous hacker and security analyst known as the Grugq had written in a blog post after the second Ukraine blackout, “This expensive light flicking makes more sense when viewed as an influence operation to signal the West that Russia has what the West itself believes are ‘real cyberwar cyberweapons.’
“Russia has flicked Ukraine’s lights twice now,” he wrote. “There is no reason to run two tests of an offensive operation if the first is successful. They want to make sure the West gets the signal.”
But Lee, who was involved in plenty of war-game scenarios during his time at the NSA, could imagine Russia striking American utilities as a retaliatory measure if it ever saw itself as backed into a corner—if the United States, say, threatened to interfere with Moscow’s military interests in Ukraine or Syria. “When you deny a state’s ability to project power,” he argued, “it has to lash out.”
Lee and his ilk, of course, had been war-gaming these nightmares for well over a decade. And as yet, cyber doomsday had never come to U.S. soil. But in the wake of Fancy Bear’s election interference, there seemed to be no limits to Russia’s brazenness. The Kremlin had meddled in the Ukrainian election and faced no real repercussions; then it applied similar tactics to the United States. Russian hackers turned off the power in Ukraine with impunity; the syllogism wasn’t hard to complete.
For John Hultquist, who had now watched Sandworm’s attacks escalate for more than two years, that next step was clear enough. Three weeks after the 2016 Kiev attack, he wrote a prediction on Twitter and pinned it to his profile for posterity: “I swear, when Sandworm Team finally nails Western critical infrastructure, and folks react like this was a huge surprise, I’m gonna lose it.”
* * *
■
On a gray day in March 2017, a taxi dropped me off in a parking lot in front of the headquarters of ISSP in Kiev. The company at the time occupied a low-lying building in an industrial neighborhood of the Ukrainian capital, surrounded by muddy sports fields and crumbling high-rises—a few of the country’s many lingering souvenirs from the Soviet Union.
When I found Oleksii Yasinsky inside, we sat down in the company’s “Cyber Lab,” a darkened room with a round table that’s covered in the same sort of network maps he’d developed for the Ukrenergo operation, long scrolls of paper showing nodes and connections of Borgesian complexity. Each map represented the timeline of an intrusion by Sandworm. By then, the hacker group had been the consuming focus of Yasinsky’s work for nearly two years, going back to its first attack on StarLightMedia. He told me there was still no way to know exactly how many Ukrainian institutions had been hit in the escalating campaign of cyberattacks; any count was liable to be an underestimate. For every publicly known target, there was at least one secret victim that hadn’t admitted to being breached, and still other targets that hadn’t yet discovered the intruders in their systems.
In fact, Yasinsky said, the next wave of the digital invasion might have already been under way even then. Behind him, two younger, bearded ISSP staffers were locked into their keyboards and screens, pulling apart malware that the company had obtained just the day before from a new round of phishing emails. The attacks, Yasinsky had come to believe, took on a seasonal cycle: During the first months of the year, the hackers laid their groundwork, silently penetrating targets and spreading their presence. At the end of the year, they unleashed their payload. Yasinsky suggested that even as he was analyzing last y
ear’s power grid attack, the seeds had already been sown for 2017’s December surprises.
Bracing for the next round, Yasinsky told me, was like “studying for an approaching final exam.” He maintained that what he and Ukraine had faced so far was likely just a series of practice tests.
He summed up the attackers’ intentions in a single Russian word: poligon. A training ground. Even in their most damaging attacks, Yasinsky said, the hackers could have gone further. They could have destroyed not just the Ministry of Finance’s stored data but its backups too. They probably could have knocked out Ukrenergo’s transmission station for longer or caused permanent, physical harm to the grid—a restraint that American analysts like Assante and Lee had also noted in my conversations with them. “They’re still playing with us,” Yasinsky said. Each time, the hackers retreated before accomplishing the maximum possible damage, as if reserving their true capabilities for some future operation. “We can only hope that they’re not done playing yet.”
Yasinsky wasn’t alone in forming that new, foreboding theory around Ukraine’s cyberwar: International observers began to posit that Russia was turning the country into a test lab, trying out digital tactics that it might later unleash on the West. Where better to train an army of Kremlin hackers than in the no-holds-barred atmosphere of a hot war inside Putin’s own sphere of influence? “The gloves are off. This is a place where you can do your worst without retaliation or prosecution,” Kenneth Geers, the NATO ambassador, told me. “Ukraine is not France or Germany. A lot of Americans can’t find it on a map. So you can practice there.”
In that shadow of neglect, Russia wasn’t only pushing the limits of its technical abilities, said Thomas Rid, a professor of strategic and military studies at Johns Hopkins. It was also feeling out the edges of what the international community would tolerate. “They’re testing out red lines, what they can get away with,” Rid told me. “You push and see if you’re pushed back. If not, you try the next step.”
And what would it look like when the hackers ceased to play those exhibition games and unleashed their full powers? In the dim back room at ISSP’s office in Kiev during my spring 2017 visit, Yasinsky admitted to me that he didn’t know what form the next attack would take. Perhaps another, more severe blackout. Or maybe a targeted attack on a water facility. Regardless, he said, he believed it would reach out, like the blackout that he felt in his own home, well beyond the internet as we’ve long understood it, into the infrastructure of the physical world.
Behind him, the fading afternoon light glowed through the blinds, rendering his face a dark silhouette. “Cyberspace is not a target in itself. It’s a medium,” Yasinsky said. “Use your imagination.”
19
INDUSTROYER/CRASH OVERRIDE
Yasinsky, it turned out, hadn’t been the only one delving into the forensic evidence of the Ukrenergo blackout. Six hundred miles to the west, another security researcher, Anton Cherepanov, wasn’t merely tracing those same footprints inside the utility’s network; he was, though neither man yet knew it, filling in the missing pieces of Yasinsky’s puzzle.
Five days after Sandworm’s December 17 blackout attack, Cherepanov opened on his computer the same set of .dll files that had represented the final, unsolved mystery in ISSP’s analysis of the Ukrenergo intrusion. Cherepanov was working in the main operations center of the headquarters of the Slovakian security firm ESET, an open-plan office with rows of workstations all facing a wall covered in screens showing visualizations of malware data feeds pulled from ESET’s antivirus software. The company, in an homage to NASA, called the room “Houston.”
ESET’s office is situated on the sixteenth floor of Aupark Tower, a corporate building that stands on the south bank of the Slovakian capital of Bratislava. The building offers a stunning view of the Danube River and, across it, Bratislava Castle looming over the city’s historic quarter. But on that day in December, Cherepanov was entirely fixated on the code unfolding on the two screens in front of him. He was working alone; “Houston” was otherwise empty. Almost all of ESET’s other employees had already begun their Christmas holidays. Only Cherepanov, as a Russian, celebrated not Western Christmas in December but Orthodox Christmas in early January.
Cherepanov had moved to Slovakia from the Russian city of Chelyabinsk in 2012 after solving a five-part reverse-engineering and cryptography challenge ESET used for recruitment. Now, looking at the .dll files at the heart of Ukrenergo’s blackout, he found an enigma as confounding as anything he’d faced in the five years since. After a combination of painstaking scrolling through the code for recognizable strings and intensive googling, he could see that the files weren’t, in fact, one payload but four distinct ones, each designed to send commands in a different industrial control system protocol—the digital lingo understood by certain pieces of electric equipment.
The code was like nothing he’d ever seen before in his years at ESET, analyzing thousands of criminal and state-sponsored hacker creations. “It was something I couldn’t understand. Most malware is simple: It steals some passwords, encrypts the drive, wipes the data. This was something different,” Cherepanov says. “I realized it’s going to be a long Christmas.”
ESET and Cherepanov had been watching the late 2016 malware bombardment of their Ukrainian neighbors from a front-row seat: The company had long sold one of the most popular antivirus programs in Ukraine, and its collection of antivirus installations had given it early access to the malware samples plaguing the country. (In fact, even as ISSP and FireEye analyzed the attacks privately, ESET had been the first to publish many of the details of Sandworm’s second Ukrainian blitz. While John Hultquist’s researchers at FireEye had classified the intrusions as the second coming of Sandworm, ESET gave the hackers behind the attack wave its own name: TeleBots, based on the Telegram-based backdoor it had first installed on victims’ machines.) So when that wave of attacks culminated in the sabotage of Ukrenergo’s transmission station a week before Christmas, ESET immediately began its own analysis of the second-ever hacker blackout.
Cherepanov refused to reveal how ESET obtained the code at the heart of Ukrenergo’s intrusion. But when he looked at the collection of .dll files it contained, they were at first as inscrutable to him as they would be to Yasinsky when the Ukrainian researcher gained access to the same code a few weeks later. Still, he could already sense their significance. He inspected the payload programs for hours that winter day, remaining in front of his screens in ESET’s silent office even after the sun had set behind the hills west of Bratislava, over the Austrian border.
Cherepanov told his wife they’d need to cancel a Serbian vacation they’d planned. When New Year’s Day arrived, he was still reverse engineering the code, digging up manuals for the obscure industrial control system protocols it used, and analyzing its functions step-by-step.
ESET’s staff returned to the office in early January, and he finally explained to them the remarkable tool of sabotage he’d uncovered: The malware was something like a self-propelled blackout bot. Once installed on a computer connected to equipment such as circuit breakers, it was designed to locate those physical machines, performing its own automatic discovery and sending configuration data back to its operators. Then, when the time came to attack, it could “speak” directly to the victim’s equipment in any of the four industrial control system protocols the .dlls contained.
In Ukrenergo’s case, only one of those four protocols had actually been used, and it had apparently opened every circuit breaker at Ukrenergo’s northern Kiev transmission station. For as long as the machine running the malware remained connected, it would keep repeating those “open” commands, in a kind of rapid-fire barrage. Even if an operator tried to close a breaker and restore the power, it would be instantly, digitally jackhammered open again.
The hackers had, in other words, created an automated cyberweapon that performed the same task they’d c
arried out the year before, but now with inhuman speed. Instead of manually clicking through circuit breakers with phantom hands, they’d created a piece of malware that carried out that attack with cruel, machine-quick efficiency.
“Holy shit,” Robert Lipovsky, Cherepanov’s boss at ESET, remembers thinking when Cherepanov outlined his findings. “This is the biggest thing we’ve worked on since Stuxnet.”
In fact, the inevitable had come to pass. One of America’s adversaries had finally built a Stuxnet of its own: the second-ever specimen of code that directly attacked the physical world.
* * *
■
ESET named the malware Industroyer, a play on its rare ability to disrupt industrial control systems. The firm knew it was sitting on a history-making discovery. But even after Cherepanov had burned through his holiday to fully reverse engineer the malware, ESET would inexplicably keep his findings a closely held secret for nearly another six months.
ESET’s staff cited a need to confirm and reconfirm their findings, a nondisclosure agreement they’d signed, and the complexities of sharing their research with Ukrainian authorities via intermediaries. It was only in June 2017 that ESET was ready to finally publish a report on the code it had found at the heart of Ukrenergo’s blackout.
On a Thursday, four days before it planned to finally reveal Industroyer in a public report, ESET’s researchers contacted Rob Lee. They wanted to give him a preview of their discovery so that the former NSA critical infrastructure security expert, who’d contributed to the most detailed write-up on the first Ukrainian blackout, could act as a credible voice to support their analysis. They cautiously sent him a portion of the Industroyer code and a draft of the blog post they planned to release.
Sandworm Page 15