According to that Heritage Valley staffer, equipment like X-ray machines and CT scanners weren’t running Windows, so they weren’t infected. But the shutdown of every Windows machine nonetheless crippled the hospitals’ operations. “The MRI didn’t get touched. But the computer that has the software to get the MRI image off the machine, that got hit,” he told me. “Tests are no good if you can’t see the damn things.”
Both Heritage Valley hospitals continued to serve existing patients, but new patients were turned away for around three days, the staffer said. The Associated Press reported that some of the hospitals’ surgeries had to be delayed. One woman, fifty-six-year-old Brenda Pisarsky, told the AP that her gallbladder surgery was interrupted by a hospital-wide loudspeaker announcement calling staffers to a “command center” to deal with the NotPetya crisis.
“Europe or somewhere in that vicinity hacked into Beaver Medical Hospital and Swickley Hospital and shutdown all their computer system! It happened right after I got into the operating room!!!” Pisarsky wrote on Facebook. “Thank God no computer was used for my type of surgery. Others weren’t so lucky and had to be cancelled.”
* * *
■
Heritage Valley’s case was an outlier. The vast majority of hospitals that suffered from NotPetya, like Sutter Health, felt its effects through Nuance’s malware outbreak, not their own. One call to deal with Nuance’s swelling transcription backlog had more than two hundred participants, Jacki Monson remembered.
In Sutter’s case, Monson claimed, the hospital network ultimately tracked down every urgent case and made sure doctors and IT staff updated medical records in time to prevent harm. “Fortunately, because of how proactive we were, we didn’t have any patient safety issues,” she said.
But not every hospital staffer was so sure. One IT systems analyst at a major American hospital—she declined to tell me which one—had a more troubling story to tell. After NotPetya’s outbreaks, she had initially focused on how to prevent her own institution from getting infected. It was only one afternoon a week later that a furious co-worker on the edge of panic had alerted her to two children’s diagnostic reports that were missing from their medical records due to the Nuance outage. Both kids were scheduled for treatments whose safety depended on their records being up-to-date. One had been transferred to another hospital for surgery the next morning.
The IT staffer felt the blood drain from her face. Did her hospital even have a copy of the dictated record changes? Would they have to delay a potentially lifesaving procedure? With only hours to spare, she located the hospital’s own raw archive of the dictations, listened to close to forty audio files, located the crucial one, and sent it out for transcription by a backup service, barely squeezing in the request in time for the child’s surgery to proceed the next day.
Over the next week, the IT staffer found two more cases where pediatric patients’ medical records were missing dictated reports, each time with only a day or two to spare before a major treatment was scheduled. In one case, a doctor had to manually retype his dictation after reexamining an ultrasound scan of a child’s heart.
In all four cases, the IT staffer told me, the hospital managed to deal with its glitches in time to prevent any delay or incorrect treatments. But even a year and a half later, she told me that those cases, where children’s care was put in jeopardy by a cyberattack, continued to haunt her. The hospital’s Nuance outage and its effects had dragged on for more than four months. Yes, the four cases she’d seen had happy endings, she told herself. But what about the hundreds of other hospitals affected by NotPetya, and their many thousands of cases? After her own close calls, could she really believe that not one among those thousands of patients had been harmed? “I can’t say how many patients were affected or what health problems might have been caused as a result of the Nuance outage,” she told me. “But there’s a huge potential for it, just by the number of reports impacted, how long they were impacted, the critical nature of the care being provided.”
If delays did occur in even a tiny fraction of those cases, the damage to human lives could have been real, argues Joshua Corman, an Atlantic Council security researcher who also served as a member of the Health Care Industry Cybersecurity Task Force. He points to a New England Journal of Medicine study that showed that even a traffic delay of less than five minutes in an ambulance caused patients to die 4 percent more often in hospitals over the following thirty days.
“Think of every hospital in the U.S. that uses Nuance. Think about how many days it was down, multiplied by the number of lab results, transfers, discharges, and how many of those are time sensitive,” Corman said. “In some cases, time matters. Pain level is affected. Quality of life is affected. Mortality is affected.”
28
AFTERMATH
One week after NotPetya’s outbreak, Ukrainian police dressed in full SWAT camo gear and armed with assault rifles poured out of vans and into the modest headquarters of Linkos Group, running up the stairs like SEAL Team Six invading the bin Laden compound.
They pointed rifles at perplexed employees and lined them up in the hallway, as the company’s founder, Olesya Linnyk, would later describe it to me. On the second floor, next to her office, the armored cops even smashed open the door to one room with a metal baton, in spite of Linnyk’s offer of a key to unlock it. “It was an absurd situation,” Linnyk said after a deep breath of exasperation.
The militarized police squad finally found what it was looking for: the rack of M.E.Doc servers that had played the role of patient zero in the NotPetya pandemic. They confiscated the offending machines and put their hard drives in black plastic bags.
* * *
■
Anton Cherepanov, working at his desk in ESET’s Houston room, had spotted NotPetya’s connection to Linkos Group’s accounting software in the very first hours of the worm’s spread. Around ten o’clock on the chaotic morning of the attack, ESET’s Ukrainian staff had sent him photos of the malware’s ransom message, and he’d quickly dug through the fresh collection of malware pulled from ESET’s antivirus software to find a sample. Taking apart NotPetya’s lightly obfuscated code, he saw that the worm was being triggered by a file on victims’ machines called ezvit.exe—a component of the M.E.Doc accounting application.
Cherepanov hadn’t dwelled on that connection. He’d been too busy trying to push out an update to ESET’s antivirus software to protect customers against the snowballing infections and then frantically searching in vain for a technique to unscramble NotPetya’s encryption or even for a WannaCry-like kill switch in its code.
It was only in the days that followed NotPetya’s initial mayhem that Cherepanov returned to the connection to M.E.Doc and began to untangle a long thread of forensic links—a thread complex enough that, more than a year later, I would have to ask him to walk me through it several times in a conference room of ESET’s Bratislava headquarters.
Cherepanov had recognized the ezvit.exe file because he’d seen it in an earlier malware outbreak. The same program had been the carrier for a different infection he’d discovered in May 2017. Five days after WannaCry, he’d found that a piece of ransomware known as XData seemed to be spreading via that ezvit executable file, using Mimikatz but not EternalBlue. At the time, he thought that victims were perhaps being tricked into installing a malware-tainted version of M.E.Doc, the sort of spoofing that hackers often use to infect victims with ransomware and other criminal code.
He’d warned M.E.Doc’s developers at Linkos Group in an email, received a brief acknowledgment, helped ESET to add protection against the new malware, and written a blog post about his findings. But in the days following the frenzy around WannaCry, few had taken notice of his warnings about an attack that had affected only a tiny fraction of WannaCry’s number of victims.
Now, a month later, he had seen M.E.Doc used to spread NotPetya, a vastly larger out
break that dwarfed his earlier findings and even WannaCry. Cherepanov had downloaded all of M.E.Doc’s 2017 updates from Linkos Group’s website when he first detected its use to spread malware in May. In the wake of NotPetya, he quickly downloaded M.E.Doc’s latest updates from May and June, just before the website was shut down by Linkos Group, and spent the rest of the week scrutinizing them. Looking at the code, he came to the realization that the hackers hadn’t merely distributed a tainted version of M.E.Doc’s software to infect victims, like a murderer serving tea laced with arsenic. They’d piggybacked on the software’s actual, legitimate update mechanism, akin to corrupting the entire tea supply of India. That remarkable supply chain hijacking meant they must have penetrated deep into Linkos Group’s servers. “M.E.Doc was itself the backdoor,” Cherepanov thought.
For the rest of the week, he pored over that corrupted update code and ESET’s malware records, working for more than twelve hours a day to understand exactly how the hackers had turned this innocuous piece of tax software into the vehicle that had carried NotPetya out into the world. It was now clear the same hackers had hijacked M.E.Doc’s updates to spread a ransomware worm at least twice, first XData in May, and then the vastly more virulent NotPetya in June.
But then Cherepanov started to make other connections back from NotPetya based on a different fingerprint—a clue that would allow him to piece together a much longer timeline.
He’d been closely following the group he called TeleBots, that others called Sandworm. as it had rampaged through Ukrainian networks with data-destroying, FSociety-themed attacks in December of the previous year—the ones that had led up to the second blackout attack. Cherepanov had since seen the group carry out more intrusions in February and March 2017. In each of those cases, in addition to that Telegram backdoor, he’d also seen the hackers install another backdoor access tool written in a programming language known as Visual Basic Scripting Edition, or VBS.
Therein lay the fingerprint that caught Cherepanov’s attention. As he investigated the M.E.Doc hijacking mechanism, one major Ukrainan financial institution—he declined to tell me which one—shared with him another, remarkable clue: Before it had been infected with NotPetya, that same VBS backdoor and another, similar VBS script had also been installed on its network via the corrupted M.E.Doc software. One of those two VBS tools seemed to be a kind of secondary foothold—designed to persist even if the M.E.Doc one were discovered and deleted. The second appeared to be a method for testing the M.E.Doc backdoor’s controls before it was used to deliver its final NotPetya payload.
Those VBS tools, Cherepanov realized, matched the attacks he’d seen for more than six months, tying NotPetya all the way back to the wave of data-destroying breaches starting in December 2016. He now saw the glowing links that chained together the entire series of incidents: It all came back to Sandworm.
Looking further in Sandworm’s history, he realized that NotPetya was, even more, a direct descendant of the KillDisk attacks that stretched all the way back to 2015. The techniques for wanton data destruction had evolved over nearly three years in the attackers’ minds. In fact, looking into its code, he could see that a list of dozens of file extensions targeted for deletion in the December 2016 attacks almost exactly matched a list targeted for encryption in NotPetya. “From the attacker’s perspective, I could see the problem. KillDisk doesn’t spread itself,” he explained. “They were testing this tactic: how to find more victims, looking for the best infection vectors.”
As he dug back through his archive of M.E.Doc updates, Cherepanov could see that they’d found that perfect carrier in Linkos Group’s accounting software. In fact, he discovered that Sandworm had first tested pushing out backdoor code through M.E.Doc two months before NotPetya, in April. The hackers had enjoyed an extraordinary level of access to the networks of every M.E.Doc customer for months, long before they’d pulled the trigger on their ultimate payload. After years of experimentation, they’d found the perfect keys into the heart of the Ukrainian internet, ideal for espionage and sabotage alike. They’d tested it, bided their time, then used it to unleash a world-shaking worm. “It was so unique,” Cherepanov marveled. “So dedicated, so patient.”
When he finally comprehended the full picture, it was 3:00 a.m. on a Sunday, and Cherepanov was still at his desk in his home office in an apartment east of Bratislava’s city center. Only the glow of his computer screens lit the room. His wife had gone to sleep hours earlier. He finished his work and got into bed, but adrenaline and the visions of Sandworm’s years-long destructive campaign continued to run through his brain. He lay there awake until dawn.
* * *
■
Largely overlooked in the chaos of NotPetya’s pandemic was a strange feature of the worm: NotPetya might not have had a kill switch or an antidote, but it did have a vaccine.
On that fateful Tuesday in late June, Amit Serper, a former Israeli government hacker with a job as a security researcher for the Boston-based firm Cybereason, was on vacation in Tel Aviv. He’d been visiting family in the suburbs of the city when he learned about NotPetya’s spread from a television news report around 7:00 p.m. Israeli time—the same time zone as Ukraine. Serper had been planning to go out drinking with friends at 10:00 p.m. “I have three hours to kill,” he thought. “Let’s play.”
Serper quickly obtained a copy of NotPetya and started pulling it apart on his MacBook and scanning its code. Within two hours, he’d stumbled onto something unexpected: an “exit process” function call. A function call is an instruction in code—in this case, one that stops a component of a program from running. Serper started working backward from that peculiarity to determine what part of the malware it might turn off and what might trigger it. Soon he came to a realization that left his mind almost numb with excitement: The feature he’d found hidden in the code could stop NotPetya’s destructive encryption altogether.
Just before he headed into the city for drinks, Serper identified the exact “if/then” statement in the code that triggered that shutdown: If a file called “perfc” with no file extension was present in the main Windows directory, then NotPetya essentially quit, saving the machine’s data from destruction. Perhaps the file was a vestigial feature of the ransom algorithm NotPetya was based on, designed to prevent the malware from encrypting data twice and rendering it unrecoverable? Regardless, if an administrator installed a file with that specific “perfc” name in that specific directory, a computer would be spared from NotPetya, like the Passover story of the angel of death sparing the firstborn sons of those who smeared lambs’ blood on their doors.
Barely able to wrap his head around the notion he’d found a possible solution to a global crisis, Serper tweeted out his finding. NotPetya had a “kill-switch,” he wrote excitedly. Perhaps it wasn’t too late to save the world from this plague after all. For the next few hours, as Serper and his friends drank beer in a Tel Aviv bar, his phone was bombarded with so many messages from security researchers, network administrators, and reporters that it repeatedly crashed.
But all of that excitement was, to some degree, misplaced. In fact, Serper hadn’t exactly found a kill switch like the one Marcus Hutchins had discovered in WannaCry. The “perfc” check wasn’t a single switch that could stop NotPetya’s progress across the world. And to have any effect, that file had to be present in computers before they were infected. That meant the task of educating victims and distributing the fix faced all the same old epidemiological problems of patching software.
No doubt some potential victims of NotPetya did end up installing Serper’s vaccine and preemptively saving their data. But by that night, when the vaccine had caught the attention of the security community, been tested, confirmed, and shared, it was too late for all but a small fraction of the plague’s victims to make use of it. NotPetya’s $10 billion worth of damage was largely, irreversibly, underway. The angel of death had already made its rounds
.
* * *
■
If nothing else, however, Serper’s work got the attention of the Ukrainian government. Cybereason contacted the Ukrainian authorities to offer its help. And following the strange, overdramatic raid those authorities had just carried out at Linkos Group’s headquarters, the cops in Kiev answered Cybereason’s staff with a unique opportunity: to aid in the analysis of Linkos Group’s hacked, confiscated servers.
The day after Ukrainian police stormed into the Linkos server room, two of Serper’s colleagues in Kiev were given access to the seized servers’ hard drives, still in their black plastic bags. They quickly copied all of the data from the machines, and Serper remotely connected to the laptops of his colleagues on the ground to analyze the Linkos logs from the company’s Boston headquarters. From around noon until late that evening, he worked to link together the fingerprints of the hackers who had penetrated the company’s M.E.Doc infrastructure to its core. Serper was so engrossed, in fact, that he continued working on his laptop even as he rode home from the Cybereason office in an Uber. Rather than waste time with a shower that night, he continued reverse engineering in the bath, perching his computer on a shelf over his tub.
Serper eventually assembled a rough story of Linkos Group’s breach: It began with the hackers exploiting a vulnerability in the content-management system of the company’s web server, the software it used for editing its website’s appearance. From there, the hackers had set up a “web shell” on the server, a kind of simple administration panel that acted as a foothold inside the computer, letting them install their own software on it at will.
Sandworm Page 22