by P. W. Singer
As Keegan and TAMS walked on, broken glass crunching underfoot, the man picked up his pace, jogging up the street. After he passed them, he got back on the sidewalk and stopped to block their path, as if he’d been standing there the entire time. “You got business here?” the man said, leaning down to put his face up close to TAMS.
“No, just passing though. Nothing for us here,” Keegan said, keeping her voice low and measured.
“Then you don’t got a reason to be here.” He spoke to her but kept his eyes locked on TAMS’s face. “Unless you’re selling . . . Or maybe I don’t have to buy.” As he said it, the man tapped the bulge where the pistol was hidden in a syncopated rhythm, maybe some song he used to motivate himself. She’d have to ask TAMS what it was later. For now, it meant that the list of ways to deescalate the situation kept getting shorter with each tap.
TAMS pushed another notice to her vizglasses. There were two men on the rooftops, also carrying weapons: AK-47 type.
Time to go. She should not have put them in this kind of situation so soon.
“Screw this,” Keegan said to the man, her voice exasperated. “You can keep this piece of shit. Not worth getting shot over. TAMS, stay here.”
With a quizzical expression, the man looked at her and then TAMS, which stood still. Keegan pinged the SUV to fetch mode and walked away to stand by the curb, as if waiting in frustration.
The whole time that she waited with her back to him, the man eyeballed her, as if waiting for the trick. When the SUV pulled up, Keegan walked around to the driver’s side and got in. At that, the man turned back to the still stationary robot, kneeling down to examine his new prize.
Keegan rolled down the passenger-side window, as if she had something to say to the man. The man looked up in anticipation just as she yelled “TAMS, follow. Now!”
She launched the SUV forward at full speed, the electric motor kicking into gear instantly, throwing Keegan back hard into the seat. As the SUV sped away, she looked in the rearview mirror and saw the robot running after at its maximum speed, about 10 meters behind the vehicle. At the intersection, Keegan made a hard right turn, the tires squealing. TAMS made a more elegant turn, leaning to the inside like a sprinter rounding the curve.
Keegan slammed the brakes and reached back to open the door behind the driver’s seat. “Get in!” she yelled. As TAMS climbed in, she checked the mirror again to see if they’d been followed. There was no sign of Glock man. The only people in view were the three kids from before, laughing and pointing. It wasn’t just that they’d never seen a robot run like that, but that they’d never seen a robot chauffeured about in the back seat of a car.
“Why does it make you drive?” the little girl called out.
Keegan set the SUV to return to the field office.
“Let’s not talk about that again,” Keegan said quietly to herself.
“OK,” said TAMS.
The Internet
Alice showed up at the door, making that very same soft knock that she always arrived with. 42 On the other side of the door, Bob thought it was Alice from the distinct signal of the knock. But despite their knowing each other for decades, Alice and Bob’s relationship was one of fundamental distrust. Some say it started when Bob let a robber just walk right through the door, mistaking the criminal as Alice.43 Others say it was when Alice accidentally hung all of Bob’s dirty laundry out in the open for the world to see.44 Whatever the original reason, the two had come to the realization that neither could be trusted. So they’d agreed to put a special kind of lock on the door. After Alice’s knock, Bob inserted a key that he alone had for his side of the door. The key could turn, but the door stayed locked, unless Alice was using her own key on the other side. Only when the lock recognized both keys, and Bob and Alice each turned them to the very same setting, could the door open.45
The two keys verified it was Alice on one side and Bob on the other, and the door opened wide. There was only one problem: it wasn’t Alice. Moving through the door was Trudy, a nasty piece of work who was wearing Alice’s skin and carrying her key.
Neither Alice, Bob, nor Trudy were actual people, or even robots like TAMS. They were enduring software characters in a digital saga that had gone on since the genesis of the Internet. Faced with the problem of how to exchange information securely on the very first computer networks, two groups of mathematicians—one working in secret for Britain’s spy agencies, the other in public at MIT and Stanford—had each independently come up with a system of cryptography that soon became the bedrock of all security in the digital world.46 Any two parties trying to exchange information, known as “Alice” and “Bob” to the cryptographers, would do so using the interplay of a widely trusted public “key” that contained an algorithm that could code and decode text and a private key, specific to that user, which allowed all other users to verify its authenticity. The system was how any “Trudy,” their shorthand for an intruder, would be kept out of the exchange.
Owned by no one person or company, any open-source ecosystem—including Professor Preston’s original software—relied on this intricate dance of human trust and digital distrust to work. The underlying software kernel that literally millions of users were sharing, for free, was always improving itself.47 Its very value was how it freely shared across the crowd everything from new features and bug fixes, to patches that closed any cybersecurity vulnerability discovered by the worldwide community of users.
But these updates weren’t just picked up and deployed by the millions of systems running the software without questioning whether it was Alice or Trudy at the door. Instead, they each went through that very same exchange of keys that had always ensured Internet security. Preston’s death didn’t change any of this; the system was designed to run without him. Bob always asked Alice for her key, and vice versa. But Professor Preston’s sudden death in Princeton did provide a moment of opportunity for Trudy to exploit, in the time between his death and his key being suspended or passed on to some other holder. The irony was that news could only be communicated out to the network through the very same system of updates. It was the briefest of windows, but one that required years of planning for Trudy to do anything with it.
While the physical swing of a cane may have helped Trudy move around one of the digital world’s most essential security checks, gaining access to the software creator’s kernel wasn’t enough. Even when Bob let Alice in, he still didn’t trust her inside his house.
To authenticate that any incoming code isn’t malware, malicious software of some type, network defenses run test after test on it, as every so often security flaws have been found even in the open-source software supposedly validated by the crowd.48 These tests hunt for revealing snippets of code that can provide a tell of some type of attack, as well as vet it against vast databases that try to find any similarities to the trillions of previously discovered versions of malware. Some defenses even run the new software though simulated versions of their network, to trigger any malware before it hits the real version. The millions of users also provide a kind of herd immunity; all a piece of malware has to do is kick off an alarm in one network, and all the others can be notified to kick it out and protect themselves.
Trudy, though, wasn’t doing anything that network defenses would deem suspicious. There were none of the normal signs that would let Bob know it wasn’t Alice who he’d let in. Trudy didn’t self-replicate, to spread from one network to another. She didn’t move laterally, harvesting passwords or user names, to log into one account after another. She didn’t collect data or open up backdoor channels of communication out of the network to try to steal information or, even worse, hold it ransom.49 She didn’t “phone home” for instruction from an outside command and control structure to allow her foothold to be exploited further.50 Nor did she trigger any of the newer tools designed to find particularly crafty malware that hid its tracks. Security researchers had discovered that for all the digital sleight of hand a Trudy might try, they
could detect something was amiss in a network by using nontraditional measures like the heat coming off their servers or the speed of the basic Raspberry Pi processors on which so many Internet-connected devices still ran. Changes in temperature or infinitesimally tiny slowdowns on these most simple of machines could give away that some unknown software was secretly riding on top. Here again, Trudy stepped right over this last trip wire, as everything she did was part of a normal software update that would cause the very same effect.
Perhaps most important, though, Trudy avoided the temptation that would have given away any other attacker who had obtained the literal keys to countless kingdoms. Despite the chance to enter millions of systems around the world, which would have been irresistible to any criminal or government, all that she was interested in was entering the door of a handful of targets. And then, once in those few, she didn’t steal a thing. All Trudy did was the digital equivalent of breathing softly on a single pair of eyeglasses left out on a table.
These handfuls of targets were mom-and-pop businesses and local governments, regulated only by the Environmental Protection Agency, whose interests lay far from matters of cybersecurity. But even if they had been manned by high-end cyber defense teams, funded by millions of dollars and staffed by ex-NSA members, Trudy was too discreet to trip the usual alarms. She didn’t race across their overall networks or alter their major operations or undertake any other actions cyber defenses typically watch for.51 When all the other billions of lines of code in the approved kernel updates coursed across the network, all Trudy did was make a tiny mathematical update to the complex equations used by a handful of sensors that existed on the edge of their systems. The only thing that shifted was a detail in the chemistry or physics that one automated sensor was observing, most importantly in the world outside the system. Moreover, that minute shift had been observed many times before, so it wasn’t viewed by the system as either novel or risky. There was always a logical explanation for it, and thus no reason to initiate an added search for anomalous behavior. And Trudy only made those new observations register well after she had erased herself, just like the very first versions of her had done when they had pioneered the dark art of digital sabotage against Iranian nuclear facilities a generation earlier.52
Each tiny shift Trudy caused at that one sensor, in one place, in one network, at one point in time, made sense on its own. At a privately owned water treatment plant in West Virginia, a conductivity measure reported back that it was detecting higher levels of selenium, which was often caused by runoff from nearby coal mines.53 The system automatically responded, as it was supposed to do, by adding in a pulse of ferric oxide to balance the chemistry.54 At a county-owned plant in Northern Virginia, slight differences in pipe pressures were reported, while at a small town-run plant in western Maryland, it was a stormwater drainage disparity—all typical findings for sensors to detect. Each initiated the standard response of deploying red fluorescent water-flow tracers.
At the Ventilation Control Facility for the Potomac River Tunnel, built by the DC Water and Sewer system to delink the city’s sewer and stormwater runoff systems that had once polluted the river after every big rainstorm, it was the air pressure setting on just four valves.55 At the US Geological Survey (USGS) stream gauge at Point of Rocks, Maryland, an hour’s drive upriver from Washington, DC, the altered sensor showed a reduction in the cubic feet per second of water flowing by, which was confirmed by similar slight differences reported by the gauge at Hancock, Maryland, another hour farther away.56 Each passed on that customary notice of a pending drought, itself a regular occurrence, to the Jennings Randolph and Little Seneca Reservoirs, used to boost Potomac River levels in times of low flow.57 Meanwhile, at the Savage River Reservoir, it was slightly higher levels of acidity in need of dilution, an occurrence frequently caused by farming and construction runoff.
To any human administrator or AI watching the activity on their own network, there was nothing digital, chemical, or physical to set off an alarm.58 Everything was operating the way it was supposed to.59 And that was exactly why Jackson Todd had sent Trudy.
Richmond (formerly Jefferson Davis) Highway
Arlington, Virginia
Keegan flicked the Petrolhead air freshener dangling from the auxiliary control stalk on the steering wheel in frustration. It gave off a faint whiff of gasoline, the smell designed to relax you by taking you back in time.
It wasn’t working, though, and neither was the music. Keegan tapped the Suburban’s dash display again and again, the force of each impact revealing her frustration. How had she allowed them to get into such a delicate situation on a training run?
A new hit flooded the car with syrupy harmonized vocals and a catchy intertwining of a rusty-sounding junkyard snare and deep electronic bass. Neural-net-written algorithmic-pop music, or A-Pop, had knocked most of the human songwriters off the charts.60 Its peppy sound clashed with the light gray of the sky, hazy from the forest fires out West.
As Keegan listened, she considered what had paired that song to the ride. Known demographic data of passenger. Past plays. Geolocation. What other nearby and like vehicles were playing. And time of day. That was all obvious. But what else? Speed of the vehicle? The number of yellow lights run or total trip time waiting at red lights? That would give the network a sense of whether it was supposed to tap into a feeling of flow or tamp down the helpless frustration that went with waiting in an autonomous car at a red light.
Then the song ended, and Keegan looked over at TAMS. “Think you could write something better than that?” she asked dismissively. Before TAMS could respond, she added, “Don’t answer that.” She switched to an oldies channel instead. It was playing “Can’t Stop the Feeling.” “That’s better,” she said, forcing a laugh. Then she remembered it would register it as a false one.
TAMS turned its head to face her. “Would you like me to change the station?”
“Why?” Keegan said.
“Your biometric portfolio reveals a sudden spike in emotions.”
“Exactly. It’s a song. That’s their purpose. You recognize it?”
“It is ‘Can’t Stop the Feeling,’ released in 2016 and associated with the film of the same year Trolls,” TAMS said.
“Yep, that’s it,” said Keegan. “It’s doing what it’s supposed to. Songs aren’t merely data, but designed to evoke, well, ‘a feeling’ that you can’t stop. It’s normal for humans.”
“OK,” the machine replied, and then provided an added observation. “However, your response is out of scope.”
A series of angry and fearful teenage faces flashed through Keegan’s mind. “Why do you think? Explain.”
“This particular song was designed to make humans happy, according to an analysis of its commercial success and online ratings.”
“The song wasn’t designed, it was written,” said Keegan. “By a human. By J.T. himself, I think. Is that correct?”
“It was co-written by Justin Timberlake, Karl Martin Sandberg, and Karl Johan Schuster,” TAMS corrected.
“Close enough. My point is that music evokes emotions. For example, we used to dance to this song at Millennial Night parties.”
It was the morning after just one of those house parties that her life had turned upside down. No. She, on the steps of the University of Washington Tacoma’s Student Center, had turned her own life upside down.
“However, your facial micro-expressions indicate high levels of unhappiness,” TAMS replied. “Have you observed anything of note in the threat environment or my performance?”
All they’d really been was a group of college kids trying to stand up to a white nationalist militia to whom the local police had turned a deliberate blind eye.61 A melee broke out at a rally for a Pakistani American student running for the state legislature, someone had died, and they’d taken the blame.62 It hadn’t been her, but that didn’t matter. A few weeks later, she’d joined the Corps, which got her as far away as humanly possible from
the arrest warrant put out for the seven unidentified people in Guy Fawkes masks.63
“No. It is just the music. Songs like this mark a moment in time or a relationship,” she said. “So, while it’s a song about happiness, it can also go with memories, good and bad ones, none of which really matter anymore. Because humans are fucked up that way.”
She realized she was doing the same thing as singing along with the radio, emoting to a machine that couldn’t care less, for the very reason that it could never start a feeling.
Keegan looked away, out the driver-side window, as they passed over the Potomac River, needing to change the subject and put some distance between TAMS and her past.
“Damn, look at that,” she said. A bright red stream stained the water, looking like an artery of blood moving through the greenish blue of the river.64 “Are there any reports on what’s going on with the river water?” she asked TAMS.
“Fourteen hours ago, the Upper Potomac River Commission Wastewater Treatment Plant reported a software problem that resulted in an inopportune level of iron oxide levels in the river water. The effect spread downstream approximately thirty minutes ago.”
“Inopportune?”
“Yes, that is the word used in their press release,” TAMS said.
“When the source is a corporate press release, translate assuming too high levels of confidence and positivity,” Keegan commanded. “Try your summary again.”
“OK. The plant reported an . . . imbalance in the level of iron oxide in the river water.”