Three Laws Lethal

by David Walton

  She slid into the car and leaned back against the seat, exhausted. With a pang of guilt, she realized she was actually relieved that Min-seo had unleashed the plague in the Mikes’ world. It had accomplished what was needed without Naomi herself having to push the button. The company could continue to thrive without her getting her hands dirty. Maybe she would have to make an effort to be nicer to Min-seo in the future.

  She arrived home, crashed into her bed, and slept much better than she had in weeks. It wasn’t until the morning, when she put on her glasses to begin the day, that she saw the news.

  The body of a young Asian woman had been found on a walking path near the east River on the Lower east Side. She appeared to have been run over multiple times by a small car. The young woman had been positively identified by police as Min-seo Cho.

  CHAPTER 19

  Brandon read the news of Min-seo’s death over his morning coffee, the story having been flagged by his glasses as one of interest. He stared at it, unbelieving, trying to convince himself he was dreaming. This couldn’t be happening. The article said nothing about Black Knight or self-driving cars, not even that Min-seo had been an employee. Maybe his cars hadn’t been involved. If one had been, though, it could mean the end. A scandal like that could destroy him.

  He wouldn’t let that happen. Not after what he had been forced to do to make this company a reality. He went to sleep every night with the image of Jillian’s face, frozen in that moment of shock and terror when he lunged toward her and she realized what he was about to do. He didn’t know anymore if she had really looked that way, or if his mind had fabricated it from his dreams. He had slept poorly since then, waking frequently to the imagined sound of screeching train brakes or wailing police sirens.

  He hadn’t wanted to kill her. The world was better off without her, but it had never been his plan. It was his father’s fault, really, with that ridiculous will. What did he expect to happen? Did he think Brandon would just wait until he was thirty before starting his life?

  It was partly his fault, too, he admitted that. He’d always had a temper. And no one had forced him to push her; he had done that all by himself. He had made a choice. He could have walked away, accepted his father’s judgment, and gotten a job at the bottom rung of some engineering corporation until he finally got his inheritance.

  When it came down to it, though, he had made the right choice. He had chosen ambition, and it had paid off. He had risked everything on a dream that he could be something more than his father had been, more than just a rich man with a big house and a nice car and a pretty young woman in his bed. A dream that he could accomplish something great.

  And nobody was going to take that away from him. He had killed to create this company. He would do whatever it took to keep it.

  He finished his coffee, rinsed the mug in the sink, and called Naomi. “Did you see what happened?” he asked.

  “I just saw it,” she said, her voice barely audible. “Our cars weren’t involved in that, were they?”

  “I don’t know.”

  “You don’t know?” A thought struck him. “You didn’t have anything to do with it, did you?”

  “What?” She sounded insulted. “Of course not! Brandon—”

  “Sorry. But you’re so secretive, and I know you didn’t like her.”

  “I didn’t like her because you hired her to spy on me. It doesn’t mean I wanted to hurt her. I saw her last night. She was heading out clubbing or something. I don’t know where, but it kind of looks like it was the Lower east Side, near the river.”

  “Did she take one of our cars to get there?”

  “I assume so. I don’t know.”

  He was getting annoyed. “Don’t you have logs? Records?”

  “Sure I do. Give me a second.” He waited. “Yeah, it looks like she got there in one of our cars.”

  “And?”

  “And what? And did it kill her? Sorry, that isn’t in the report.”

  “We have to ground them,” he said, angry at her tone. This was serious, didn’t she see that? “We have to send all of our cars back to the parking garages and check them one by one.”

  “I know which car she took.”

  “Ground them all! Until we understand this, I don’t want any of them on the road.”

  “Okay,” she said.

  “Which lot is the one she rode going to?”

  Another pause while she looked for the answer. “Mount Vernon.”

  “We’ll start with that one. I’ll meet you there.”

  The Black Knight parking garage in mount Vernon, like the others scattered around the city, represented a compromise between the cost of the real estate and its distance from the cars’ main cruising grounds. That generally meant placing them in poor areas, undesirable for development, which also meant substantial physical security. There was little chance the cars could be stolen—all the steering wheels were deactivated, for one thing—but they didn’t want them vandalized, or the space used as temporary housing or for drug deals.

  Brandon met Naomi at the gate, and they walked through together. He was starting to think that grounding the whole fleet had been overly conservative. After all, she’d been walking around the Lower east Side on a Friday night. There were bars and clubs all over that neighborhood. Chances were she’d been hit by a drunk driver.

  When they reached the car that had transported Min-seo the night before, however, his hope that it would have nothing to do with his company vanished. “Look,” he said. The fender was dented slightly and streaked with red.

  “No,” Naomi said. “Oh, no.”

  “You should leave,” Brandon said. “I’ll clean it up. You take care of the digital side. If there are any tracks to be covered in the computer system, cover them.”

  “We can’t do that. We have to call the police.”

  Was she an idiot? “Are you kidding? That would destroy us. Killer robot car? We’d never get a fare again. The city would revoke our license.”

  “But Brandon, we killed her. Our company, our software. We can’t just cover it up. What if it happens again?”

  That made him angry. “Your software. Your software did this. Don’t try to pin this on me. If you call the police, it’s going to be you taking the fall.” He pointed a finger in her face. “If you don’t want it happening again, then fix the problem.”

  Naomi took a step back. She looked scared, but determined. “She has a family. People who cared about her.”

  “So what? Destroying the company isn’t going to bring her back.” He paused and took a breath. She just wasn’t thinking of this the right way. “They’ll take your software away from you,” he said.

  That struck a chord. He could see it on her face. He knew what really mattered to her. And he was right; if the police got involved, they would want to examine the software that commanded the vehicle. If the company went under, she would lose it altogether.

  Naomi touched her glasses. Brandon took a step closer, incredulous. “What are you doing? Are you calling the police?”

  “You know it’s the right thing to do,” she said.

  Hot rage filled his chest. Who did she think she was? This was his company. He had pulled her off the streets when she had nothing. He reached out to knock her hand away, but his open hand caught the side of her head and knocked her glasses off. They clattered to the concrete. She reached down to pick them up, but he kicked them, sending them skittering out of reach. She looked up at him with shock and outrage in her face, and he lost control. She was supposed to be on his side. He grabbed her by the hair and pulled her to her feet again, then held her face roughly with his other hand. “This company is mine. Nobody’s going to take it away from me.”

  He wanted to kill her right then. To smash her face into the car and erase that look of righteous goodness forever. Who was she to tell him what was the right thing to do? But he held himself back. As annoying as it was to admit, he needed her. Killing her wouldn’t serve his pur
poses.

  “Okay,” she said, and the terror in her face eased his anger. “I’m sorry.”

  “This is not your choice to make.” He pulled her to the side, still gripping her hair and her face. She shrieked, not knowing what he was going to do. He forced her to walk toward the bloody car. When they reached it, he let go of her face and took her right hand, twisting it around and pressing her fingers under the handle on the driver’s-side door. Leaving her fingerprints. “Just a little insurance policy,” he told her. He yanked a few hairs from her head and, still using her fingers, opened the door and tossed the hairs into the driver’s seat. Then he looked at her, satisfied. “You stay quiet, nothing happens,” he said. “You talk, and I’ll see you go down for murder. I know you hated her. She was taking over your job, and you killed her out of jealousy. Trust me, it’ll stick.”

  Finally, he let go, and she lurched backwards. She touched her head where he had torn out her hair. He watched her closely. Would she run? But she just stood there, apparently submissive. Good.

  “Go back to the office,” he said. “Find the evidence and delete it. I’ll be back in an hour, and you’d better be able to convince me it’s gone. When the police ask, Min-seo was at work during the day as usual. That’s the last time you saw her.”

  Naomi nodded. “Good,” he said. “Now go.”

  Naomi walked back into the office like a sleepwalker, but inside she was frantic. Everything around her felt too sharp, too solid. It ought to be a dream, but it too clearly wasn’t.

  Brandon had just attacked her. He hadn’t hurt her in any lasting way—she probably wouldn’t even have any bruises—but she had never seen him that angry. He terrified her.

  What should she do? The brave choice would be to call the police and take her chances. But she could still feel Brandon’s hand bruising her cheeks, feel his breath on her face, and she knew that she wasn’t brave.

  If she went to the police, he would win. She had never been good at talking. He was charming, convincing, good at selling his view of things. She would mumble and get nervous, and it would all come out wrong. Besides, what could she prove? They would believe him, not her, and then he would get his revenge. Maybe she should just run away. Get as far away from Brandon as she could. She could move to a different city and lose herself in the anonymous crowds. But she couldn’t do that. She would be leaving Brandon in charge of software that could kill, with no expectation that he could find the problem or make the right choices. She would be putting more lives in danger.

  And then there were the Mikes. If she abandoned them, or if the company went under, they would die. Not just the fraction that Min-seo had killed, but all of them. Possibly she could start over, find some other benefactor willing to pay for her work, and try to create their world again, but would she be able to recreate what she had done here? She didn’t know. And whether she could or not, all these Mikes would be dead.

  Before she made any decision, she had to investigate and understand what had happened. Then, at least, she would be deciding based on knowledge. After that, if she wanted to disappear or call the police, she would still have that chance.

  She scoured the logs, looking for the details from the time of Min-seo’s death. She located the specific car that had picked her up and the specific Mike that had been driving it. The murder, she found, was hidden in plain sight, captured in the record of the drive, now that she looked more closely. Min-seo had used the Black Knight app to request transportation to a bar two blocks from the place her body had been found. She had never arrived. The car had delivered her, instead, to an empty pier where it had stopped and announced—in the pleasant female voice that all of their cars employed—that she had reached her destination.

  Naomi imagined Min-seo, confused, looking out the window at the dark pier and the water beyond. According to the log, she used the app again to renew her destination request, but the car continued to insist it had arrived. She must have realized that the bar was easy walking distance away and, frustrated at what seemed like a software bug, climbed out of the car to walk. Which was when the car had run her down. Naomi could see it in the log: forward and reverse, then forward and reverse again. A cold record of murder. The car then smoothly slid away and stopped at the bar anyway, registering successful completion of its trip.

  There would be video, of course. The cars practically bristled with cameras, the feeds recorded and transmitted and saved. Naomi had no desire to actually see it happen, however. She didn’t know how quickly Min-seo had died, and she didn’t want to know. Her imagination was bad enough. When she looked for the video records, though, she found that they were already gone. Deleted. Brandon couldn’t have done it; even if he knew where to look, he wouldn’t have had the chance. Yet they were gone.

  A few hours later, Brandon returned to the office, his face pale and his expression grim. He leaned over her desk, and she felt her heart rate quicken. “Are we good?” he asked.

  She kept her eyes down. “All taken care of.”

  He sighed. She risked a quick look up, and saw his face soften. “I’m sorry about before. Did I hurt you?”

  She lowered her gaze again and made a noncommittal noise. “You forgive me, right? It’s just that you made me so angry. We’ve put our whole lives into these cars. We can’t just throw it away for no reason. If it could bring Min-seo back, I’d do it in an instant. You know that, right?”

  She didn’t answer. He scowled again and stalked to his desk, where he sat and stared out the dark window at nothing.

  Naomi returned her attention to the computer. The question was no longer whether the Mike had killed Min-seo, but why it had. And the answer was obvious. Because Min-seo had run the script that had massacred millions of them. The Mikes had been evolutionarily shaped over millions of subjective years to survive. They had developed the sophistication of societal hierarchies and innovative technologies. Was it not reasonable to believe that, recognizing a threat that presented such a danger to them, they would eliminate it?

  If true, however, it meant three shocking things that Naomi had not realized. First, the Mikes were aware of humans as independent beings. That was not necessarily too surprising; ever since Naomi had connected them to cars in the real world, the Mikes’ survival had been linked to the pattern of the movements and choices of humans. Second, the Mikes understood that an individual human who rode in a car could be the same human who interacted in specific ways in their world. That wasn’t too hard to see either—min-seo would have logged in whenever she wrote software or made changes in the simulator, and her account on the Black Knight driving app was linked to the same employee account, allowing her to use the transportation for free. Third, and most chilling of all, the Mikes recognized that just like themselves, humans could be terminated. If Naomi had initiated the script, would it have been her broken body discovered on the road?

  “How did this happen?” Brandon suddenly asked, his voice exploding into the still room. Naomi startled. She had almost forgotten he was there.

  What could she say? That Min-seo had done something the software didn’t like? That it had killed her out of self-defense?

  “It was a bug,” she said. “A bug?” he shouted, his voice shaking with rage. “A bug? Was this the same bug that killed Abby? Do you even understand what’s happening here?” Spit flew from his mouth, and his fists clenched.

  “I don’t know,” she said. “I’m doing the best I can.”

  He snatched a stress ball from his desk and began to squeeze it ferociously. “This is the second young woman your software has killed. I think that qualifies as more than just a bug.”

  Abby. Of course. Naomi’s mind spun. She hadn’t even made the connection to her sister’s death until that moment. They couldn’t be related, could they? Abby’s death had been so long ago, from a software perspective. That had happened before the Mikes had expanded their world, before she had trained them to drive cars from their control booths. The Yotta demo cars had be
en driven by a copy of a very old Mike running as a simple machine learning algorithm, external to the Realplanet world. It had been a hugely simplistic early version compared to what the Mikes had accomplished since.

  Besides, Abby had died because of a hack. Hadn’t she? They had never discovered the hacker, but Tyler had been convinced it was the only explanation, and Naomi had believed him. It had to have been the work of a human, because of the physically disabled kill switches. Besides, Abby hadn’t contributed to the software, never mind run a script to initiate a massacre. What reason would the software have had to kill her, even if it could?

  Then she remembered. When Abby was killed, they had been about to start the part of the demo in which she would climb into a car and drive recklessly, forcing the other cars to swerve out of her way. To make it work, they had disconnected that car’s Mike from the controls. If they didn’t, it would self-correct Abby’s maneuvers for the safety of all the vehicles. Which had, of course, been its core principle: the safety of the vehicles. Mike had wanted to keep the cars safe, and Abby had prevented it, forcing Mike to stand by while its car was endangered and the others swerved desperately to avoid an accident.

  What if it had found another way to keep them safe? They had practiced the demo over and over again, so the pattern would have been easy to predict. Faced with yet another reckless drive by Abby, the Mike had taken steps to prevent it. She remembered the ruthlessness with which those early Mikes had fought for resources, the tyrants that had risen in their communities, and how quickly they had killed others like themselves to survive. The Mikes had no moral compass, no sense of right or wrong. They acted according to the behaviors that the need to survive had developed in them over generations. They could have done it. They could have killed her.