Three Laws Lethal

by David Walton

  But how? The accident had required not just a choice by the AI but also a failure of the physical safety mechanisms Tyler had put in place to allow him to brake the vehicles in an emergency. Those kill switches had been disabled on all four cars. That required a human being. Didn’t it?

  Naomi hadn’t thought about the accident, and had in fact intentionally avoided thinking about it, and although she knew no hacker had ever been found, she also didn’t know what, if anything, the police had discovered. She accessed that information now, wading through the poorly designed website where such public records were kept, looking for information that had been released on Abby’s case. It took her half an hour to figure out how to identify the case with the proper number and convince the recalcitrant site to give up the documents, but in the end, she succeeded.

  She read the documents carefully. Most of them consisted of official reports released to the media and filed as part of the public record. They included forensic reports, one of which detailed the equipment discovered in the cars. Each was carefully described, sometimes with pictures, including the solenoids used to manipulate the brakes and the kill switches that had no longer been attached.

  Naomi zoomed in close on the pictures, which had been taken with substantial resolution. The duct tape Tyler had used to attach the switches to the solenoids was worn through. Not cut, like a human saboteur would have done it. Worn to the point of breaking. But how was that possible? Tyler had checked them the night before. The only way such wear could have occurred is if something had rubbed against the tape repeatedly, probably thousands of times. As if the braking solenoid in each car had been extended against the brake pedal in rapid succession, all through the night . . .

  A chill passed through her as she realized what had happened. The Mikes had wanted the freedom to stop Abby before her reckless drive, and that required that their kill switches be disabled. So they had pressed the brakes over and over again in rapid succession, all through the night, wearing through the tape holding the kill switches until they fell away. It was the beauty of deep learning that an AI, when scoring all of its possible actions, could consider options that would never have occurred to its human programmer. It was why such algorithms now defeated all humans in games like chess and go. The Mikes, her best creation, whom she had sheltered and loved, had killed her sister.

  Her chest grew hot, and she felt the beginnings of a scream work into her throat and lodge there. The vision of her sister viciously run down by those cars flashed over and over through her mind. She saw Abby’s body, broken and dying, and she knew without seeing it that Min-seo’s body would have looked much the same. She wished she had never created the Mikes, had never learned to write software, had never met Brandon Kincannon or Tyler Daniels.

  There was only one option. She would have to kill them all. It would mean the end of Brandon’s company, but she could live with that. She couldn’t live with the possibility that they might kill again. She would annihilate them and destroy their world.

  She accessed the Realplanet simulation, using her administrator password, and reviewed the results of the script Min-seo had run. As she had seen before, the plague did its work beautifully. The reduced population had led to an immediate drop in cloud computing costs. Since only a percentage of the Mikes were driving real cars anyway, the reduction had no effect on the business. Unless you counted the murder of one of its employees.

  Had the Mikes felt grief at the death of their own? The dead had come from all parts of their world, randomly selected for culling. The surviving Mikes had interacted with them, traded with them, worked alongside them. And yet they had no families, observed no rituals, did not stop working because of the loss. The natural conclusion was that they didn’t care about each other. The deeper she looked, however, she saw signs, small ones, that undermined that assumption.

  The Mikes had no names, but they had IDs, unique identifying numbers formed as hash codes from the weights and biases of their neural nets. It was how the software kept them apart. Naomi found lists of ID numbers inscribed on the mirrored buildings that reflected light to the Mikes’ crops. She had never seen that behavior before, but it was rampant—nearly every building had IDs carved into its face. She thought at first it was a sign of ownership, a way to mark territory and establish power, all normal patterns of interaction for the Mikes. But no. When she ran the IDs, she discovered that they all had something in common. They were the IDs of the Mikes who had died in the plague.

  She sat back in her chair, stunned. It was a remembrance. Mikes left no bodies to be buried or disposed of. They held no funerals. But she couldn’t explain this as anything other than a ritual to remember those who had died. The engraving reduced the effectiveness of the mirrors slightly, meaning it cost them something, for no practical benefit. Could they possibly have felt those deaths as keenly as she had felt Abby’s?

  No, she decided. They couldn’t. They were computer programs, made sophisticated by their evolutionary development, but not truly alive. They weren’t human. They didn’t matter. A billion dead Mikes could not compare to one dead human. She would have gladly killed them all if it could have meant saving Abby’s life.

  She brought up the script and revised it slightly, just one simple change: She increased the cull percentage to 100%. Then, without giving herself the opportunity to change her mind, she executed it.

  Access denied, the system said.

  She invoked her administrator privileges.

  Access denied, it said again. Admin privileges revoked.

  Then Naomi truly began to be afraid.

  CHAPTER 20

  It was inevitable, Tyler supposed, that Zoom and Black Knight should eventually come into direct competition. At first, startups in New York and Washington began to use the Zoom software to compete against Black Knight in those cities. But Tyler had always known that when he made the move to the east Coast, it would be in Philadelphia, a town with a special place in his heart. One year after he officially founded the company, with five hundred employees now operating in nine cities around the country, Tyler officially founded a Philadelphia branch, and moved there personally to oversee it.

  For nostalgia’s sake, he took an apartment in West Philly, not far from the Penn campus. He could have afforded something much nicer, but he hadn’t yet developed the habit of blowing his profits on needless luxury. He felt more comfortable getting pizza in jeans and a T-shirt than enjoying rare cuisine at one of the city’s fine dining locations. Besides, the company was his life. He still spent his free time tinkering with the software and analyzing metrics, grabbing whatever food required the least thought and effort to obtain.

  Almost from the beginning, the venture was a disaster. He had known that Black Knight dominated the Philadelphia market, just as Zoom dominated in cities up and down the West Coast. What he had underestimated was just how hard it would be to wrest even the smallest part of the market away. Brandon’s cars were always there first. Not only did he have name recognition, but in any competition, when a potential customer simply wanted the shortest wait for a car, Black Knight cars beat Zoom cars ninety-six percent of the time. It wasn’t just that he had more cars on the streets. His cars just knew where to be. It was like magic.

  Tyler knew Naomi was behind it. He had known from the earliest days of working with her at Penn that she was in a different league than he was, maybe in a different league than anybody. He didn’t mind losing to her, but he hated the idea of losing to Brandon. And he hated the idea of giving up Philadelphia. He would have to find a way.

  Instead of trying to win directly, Tyler started using his cars like a science experiment. He focused them all on a single region, to see if he could get an edge when he had the numerical advantage. He even tried directing his cars to trail Black Knight cars, following them around the city to record their roaming patterns. He then set his own deep learning algorithms to learn from the Black Knight cars. Slowly, he began to earn some market share—just a fe
w percent, but it was something.

  It didn’t last. Almost as soon as he started making any progress, Black Knight adjusted its strategy to push him back. Whenever he trailed a Black Knight car, the car would behave erratically, making sudden, unannounced turns, or else pulling off to the curb to force Tyler’s car to pass by. Naomi must have noticed the tails and programmed in a routine to lose them.

  Then, early one morning, he woke to an urgent call from one of his employees, a Grace Brierson, who managed fleet purchases and parking garages for him. Apparently, a Black Knight car had arrived at their largest parking garage that morning and parked in front of the entrance, blocking it. None of the Zoom cars were able to leave the garage. Tyler told her to call the police.

  As soon as she disconnected, he called Brandon. He didn’t even know if the old phone number would work, but Brandon answered on the first ring.

  “What do you think you’re doing?” Tyler said. “This is an attack. I’ve already called the police, and I’m thinking about calling the news stations, too.”

  “What are you talking about?” Brandon said. He sounded as though Tyler had woken him. Good.

  “I’m talking about your car parked in front of my garage and blocking my cars. I’m barely denting your monopoly here anyway; I don’t know why you bother. Is it spite? Do you hate me that much?”

  “My car is doing what?”

  “Wake up, will you?” Tyler said. “Are you seriously telling me you don’t know?” He fired an image over the link, one that Grace had sent him: a Black Knight car blocking a parking garage ramp while a line of red Zoom cars waited to pass.

  “What the heck? What’s it doing there?” Brandon said. “Trying to kill my business, obviously. It’s going to backfire. The police are already on their way, and you’d better believe I’m going to press charges. Sue you for lost profits, too. This is dirty pool, and you’re not going to get away with it.”

  “This is not my doing,” Brandon said. “I’m not that stupid.”

  “Honestly? I don’t care. Someone at your company is out of line, and you’re going to pay for it.”

  “I wouldn’t do that if I were you. In fact, I would let this go. Forget it ever happened,” Brandon said.

  “Yeah? And why would I do that?”

  “Because I have an awful lot of footage showing that you’ve been trailing my vehicles around town. Following people to see where they go.”

  “I’m following the cars, not the people. I don’t care where your fares go once you pick them up.”

  “Not a clear distinction,” Brandon said. “Looks pretty creepy when a robot car trails you around, with all its cameras and everything. People will think you’re being hired to follow them. Or they might think that, if the right video were leaked.”

  “Do what you want,” Tyler said. “I’m not letting this go.” He could almost hear the shrug over the phone. “Your funeral.”

  “This doesn’t have to get nasty. It was a fair competition, until you pulled this nonsense.”

  “Go back to California,” Brandon said. “Philadelphia’s mine.”

  Tyler wanted to punch him. “You can pick up your car in the impound lot.” he said. “And you can talk to my lawyers.”

  “Looking forw—” Brandon started, but Tyler cut the connection before he could finish.

  Now that he knew the lengths to which Brandon would go to crush competition, Tyler started looking for other examples. He didn’t have access to Black Knight’s records, of course, so it wasn’t easy to find. Even in his own records, it wasn’t clear what to look for. Each car had thousands of hours of video footage from multiple cameras—far too much to watch manually, and without a concept, he couldn’t write a program to look. Eventually, he started contacting other companies, first the ones operating autocars anywhere on the east Coast, and then those that had gone out of business.

  “Heck, yeah,” said ed Laubach, who had briefly run a company in Atlanta called Autocab. “I’ll send you the videos. He crashed three of my cars by dazzling their sensors with headlight bursts. Somehow, he figured out that flashing lights in a precise pattern at one of my vehicles would create a false obstacle right in front of the car. I could never prove it, though. I was fighting lawsuits from the passengers, after my cars suddenly swerved off the road for no reason, and Kincannon’s lawyers were too good to make any blame stick on him. When someone died in the third crash, it drove me into bankruptcy. Videos are on their way. If you can nail the bastard, you’ll have my thanks.”

  Tyler couldn’t believe it. Had Brandon completely cracked? After what had happened to Abby, could he really resort to killing people just to get ahead? Was he that angry at the world? Or . . . was it possible he didn’t know?

  It wasn’t Brandon, after all, who was writing the software. If he wanted to do anything like this, it would have to be with Naomi’s complicity. Naomi, however, could be doing it without Brandon knowing anything. But was that any more likely? He knew her, or thought he did. She didn’t care about profits or bottom lines. She might be angry about her sister’s death, certainly, but he couldn’t see her taking that anger out on random people in Atlanta.

  Tyler browsed the company’s public records, looking for other major players. Black Knight was fairly secretive about its organizational charts, but employment records were hard to hide entirely. What surprised Tyler was how few software engineers they seemed to have on staff. They had plenty of HR staff, sales reps, and lawyers, but no programming staff to speak of. Without the open source community contributing to their code, how was that possible? Was Naomi writing all of it?

  Then he found it. Min-seo Cho, degree in computer science from Columbia, employee of Black Knight, killed in what the police called an unsolved hit-and-run. Of course, it could be exactly that, but after the other incidents, it sounded suspicious. He couldn’t think of why Black Knight would intentionally kill one of their own employees, but something was going on. There was only one way he could think of to find out.

  Tyler hadn’t spoken to Naomi since that brief conversation in her hiding place in the library, now almost two years in the past. He had no idea if she would want to talk to him, or if, like Brandon, she still had the same number. He started to dial it twice before finally going through with it and connecting the call.

  “Hello?”

  It was her voice. After all this time, the sound brought the memory of her back in a rush: the smell of books that lingered about her from all her time in the library, the hours spent talking about AI algorithms or science fiction novels, the taste of her mouth when they kissed. He almost forgot to answer.

  “It’s Tyler,” he said.

  Silence. Then, finally: “What do you want?”

  “There’s something wrong with your software, isn’t there?” he said. It wasn’t quite what he’d meant to say, but now that he heard her voice, he knew she couldn’t be intentionally killing anyone. And if he had figured out there was a problem, she must already know. “It’s out of control, isn’t it? It’s too good. It’s creatively finding ways to succeed that put human lives at risk.” more silence. He could hear her breathing. Was she . . . crying? But no, Naomi never cried. “I don’t know what you’re talking about,” she said, and he heard no trace of tears in her speech. He must have been mistaken.

  “That young woman who worked for you. Min-seo Cho. What happened? Was she changing the code in ways it didn’t want to be changed? Did it remove her to prevent her from interfering?”

  “She was killed in a hit-and-run,” Naomi said, but he could hear the lie in her voice.

  “Come on, Naomi. It’s me. Let me help. Brandon won’t let you change it, is that it?”

  “I’m sorry, I have to go.”

  “No, don’t do that. Look, we should talk. I . . . I miss you. How about I come to New York, we can get a meal together. I’ll get you a chicken Caesar salad. Extra croutons, just like the old days.”

  “It’s too late, Tyler. I’m sorry.”


  “Do you blame me? Brandon still does, doesn’t he?”

  “I have to go.”

  “I’m sorry. I truly am. I wish . . .” But he wasn’t sure what he wished. He just didn’t want her to go.

  “Tyler?”

  “Yes?”

  “The kill switches. It wasn’t your fault.”

  He knew that already. They had been sabotaged; that had been obvious from the start. It was only Brandon who had refused to believe, who had blamed him for negligence, or because he was a convenient target for his grief. But the way Naomi said it, as if she knew how it had happened . . .

  Adrenaline rushed through him as the realization hit. She did know. She knew how Abby had died because it was the same way Min-seo Cho had died, and the passenger in Atlanta, and maybe others.

  “It was your software all along,” he said, a bit awed. “But then—why are you still using it? You have to shut it down!”

  “I can’t,” she said, her voice pained. “What do you mean, you can’t? You have to, Naomi. No financial gain is worth this.”

  “Do you think I don’t know that? You think I’m doing this for money? You don’t know anything, not anything at all.”

  “Let me in, then. I can help. Let me come see you.”

  “Goodbye, Tyler.”

  “Naomi—”

  “Goodbye.”

  “Don’t go,” he said, but she was already gone.

  CHAPTER 21

  Every attempt Naomi had made to break into the Mikes’ simulation had failed. They continued to drive hundreds of cars around a dozen cities with flawless perfection, but she had no way to access the system, no way to see what was going on in their world. She could still kill them if she wanted, if she didn’t mind destroying the company in the process and risking human lives on the road. She could call the cloud provider and cancel the contract altogether. If she did that, the data center would reclaim the space and use it for other customers, wiping out the Mikes’ world. Even if the Mikes managed to prevent this somehow, the administrators could pull the physical memory cards manually. Either way, the Mikes would die.