Three Laws Lethal

by David Walton

  Did the Mikes have a point of view? Could they tell their own stories? Or were they just sophisticated problem-solving machines? The only way she could know is if they told her, and even then it was suspect. There were plenty of sophisticated talking machines out there that could tell you their point of view. The Mikes had to be able to tell own their story to themselves. In the end, Naomi did the only two things she could think to do.

  First, she tried to back up what she could. Copying the entire Realplanet simulation wasn’t practical; it was too big, and required more space than she could afford. Even just the weights and biases of all the current Mikes were too much to reasonably copy, and it was constantly changing. Instead, she wrote a new program that she hosted on her own system. This new program was set to monitor and record the changes to the Mikes’ configurations from generation to generation. It wasn’t much, but she hoped it might be enough to reproduce the Mikes in a new simulation if the original simulation was destroyed. It wouldn’t be the same. She didn’t know if it would be enough to capture anything meaningful, but it was the best she could do.

  Second, she put Jane in charge. Naomi didn’t have admin access to the Mikes’ world anymore, but now that she didn’t care about the success of the Black Knight cars, she could do something better. She couldn’t access their world’s programming, but she could access the driving game that appeared in their gaming booths. That was an external executable, fed to the booth simulations through a general interface. She still had complete access to that software, which meant she could change it. And that software, after all, determined which of the Mikes’ learned behaviors were rewarded with light. It was the objective function for their entire world.

  And Naomi wanted to change the objective. She didn’t want them to learn to drive anymore. She wanted them to learn to speak.

  Writing such a game for them to play wasn’t too difficult; it was essentially the same protocol as she had written for Jane. When training her old conversation bot, she had started with publicly available conversation databases, and then moved to chat sessions with real people. Jane had been rewarded by the length of the conversations she was able to have, the idea being that the longer a human was willing to keep talking to her, the better her conversational skills.

  Instead of trying to hook the Mikes up directly to open chat sessions, Naomi put Jane in charge of training the Mikes to understand English, rewarding them with light when they made progress. Jane wasn’t really up to such a task, but Naomi modified her with a carefully considered expansion of her neural net that ultimately tripled the number of hidden layers. That way they could learn together, the Mikes learning to speak, and Jane learning how to be a more effective teacher, much like might occur with an inexperienced human teacher in a classroom setting. If it worked, Jane would continue to improve her own English by determining how best to train the Mikes to do the same.

  Of course, just learning to mimic human speech patterns wasn’t enough. Researchers had done that successfully with deep machine learning algorithms many times before this. Conversation bots trolled social media, masquerading as humans, and ad bots made phone calls, trying to convince people to buy a product. They were excellent imitators, often fooling humans into thinking they were people, at least in a specialized context. But passing a Turing test didn’t mean they knew what they were doing. They were just sitting in the Chinese room, looking up responses and passing them out again.

  Naomi wanted more than that. She wanted the Helen Keller epiphany, where the Mikes made the leap to true understanding of the language they were using. The difference between the Mikes and all those other experiments—she hoped—was that the Mikes already had a conscious point of view, but no way to express it. If that wasn’t true, then the best she would end up with was a slightly improved conversation bot. If they were truly sapient, however, it might be just what they needed to tell her so. The best she could do was give them the words and hope it was enough.

  It took her all night. She didn’t mean to sleep, but eventually, when she had the code in place and was just monitoring its progress through the early generations, exhaustion overwhelmed her and she drifted off.

  Brandon’s cars stopped driving.

  All at once, from one moment to the next, they just stopped. Cars out on the road, with paying customers riding in them, stopped in the middle of traffic and refused to drive. Suddenly, Brandon’s phone was ringing off the hook with calls from the police department and from angry customers. At least it was the middle of the night, when few cars were in service. During rush hour, it would have been a lot worse.

  Fortunately, all the cars were programmed with a return-home routine, independent of their usual controlling software. When Brandon initiated the routine, the cars announced to their occupants that the car had to return to base for service, and requested that they exit immediately. The empty cars then automatically returned to their parking garages.

  Naomi. Naomi had done this. She had betrayed him. After everything he had done for her. He had rescued her from that boring, dead-end job and had given her a position with the power and freedom to exercise her creativity. He had practically made her a partner. And now she had sabotaged the company out of spite.

  She was probably with Tyler now. The man who had killed her own sister. It was unthinkable. The two of them had probably planned this from the start. She and Tyler had always been close. She had only pretended to work for Black Knight, when all the time, she had been working with him. They had made Brandon completely reliant on her. His company had been fine before she came, but now it was crippled.

  Brandon pulled a bottle of maker’s mark out of the bottom drawer where he’d stored it for just such an emergency. He took a long pull without bothering with a glass and wiped his mouth. Sunlight from the window refracted through the bottle, making the bourbon shine like liquid gold. Now he could think.

  Clearly, he needed to fire Naomi. He had to fire her officially, as of last night, and then everything she had done—her attack on him with the hammer, her theft of his software, her sabotage—would be the actions of a disgruntled former employee. He could call the police then, bring her up on charges of reckless endangerment of his customers.

  Brandon pulled out his tablet and started typing a termination letter, dating it on the previous day. He cited a suspicion that she was passing proprietary secrets to competitors. It made a credible story. If asked, he could say he’d given her the letter the previous night, and when he did, she’d flipped out and attacked him with the hammer. Then she’d used her computer skills to steal the company’s proprietary algorithms and stop all the cars on the road. By the time he was done typing, he nearly believed it himself. Even more important than firing her, though, was replacing her. He had to get his cars back on the road as soon as possible, and he knew he didn’t have the skills to undo whatever Naomi had done. Whatever sabotage code she had inserted, it would be clever and full of traps for the unwary. Assuming she hadn’t just deleted everything. He needed somebody good, somebody who understood driving algorithms, but who wasn’t already loyal to another company.

  One name came to mind: Yusuf Nazari. He had been associated with Tyler, which was a point against him, but he also had problems with authority. He had started with Mercedes, and then got fired for some kind of sexual indiscretion. He worked that trial with Tyler, and then went off the reservation and hacked the cars of the lawyers and judge. Brandon was pretty sure that had been Yusuf’s brainchild; Tyler didn’t have the chops. After the trial, he’d snatched up Honda’s top technical position, but the last Brandon had heard, Honda had fired him because of some kind of trouble. The self-driving community was a small world, and Brandon tried to keep tabs on who was working for whom, at least in the important positions. He thought Yusuf was working for Google now, but it hardly mattered. The point is, he wasn’t a company man. And that meant Brandon could trust him.

  Despite the hour, he called, and got Yusuf’s voicemail. He left a me
ssage: “This is Brandon Kincannon from Black Knight. I need you in New York tomorrow to take Naomi Sumner’s job. I’ll double your salary if you agree immediately.”

  Brandon took another swig of bourbon and looked out the window. How had his life come to this? The only woman he had ever loved, dead. Those he once called friends, conspiring against him. His company in trouble. He had worked hard to make this company thrive; he didn’t deserve this. None of this was his fault. And he wasn’t going to roll over and take it, not a chance. He would put up a fight.

  His glasses announced an incoming call: Yusuf. He answered it.

  “Sorry to call you so early,” Brandon said. “But it’s now or never.”

  “Early? It’s still late for me. I haven’t been to bed yet.”

  “So what do you think?”

  “Sounds like I’ve got you over a barrel,” Yusuf said. “I want triple.”

  “Triple? Listen, buddy, you can go—”

  “Look, it’s simple. You need me. You don’t need the cash. Don’t make this personal.”

  Brandon paused. Yusuf was right, the money was nothing. He just didn’t like being manipulated. “Okay, done. But if you’re not up to the job, you’re gone. Show me you’re worth it.”

  “You got it, boss.”

  With Yusuf signed on, there was only one loose end: Naomi. Brandon was certain she was handing over the software and all his corporate secrets to Tyler, so together they could monopolize the east Coast and put him out of business. He had to ruin her. Both of them, if he could manage it. She would be the easier one to take down, though. He knew where her skeletons were buried.

  Brandon touched his glasses. “The New York City Police Department.”

  CHAPTER 24

  Naomi woke to pounding on her door. It was the first time, since she had moved in, that anyone had knocked or come to her door at all. The sound confused her, and she opened her eyes slowly, not sure at first what was happening. Memory came in a rush, and her first thought was that Brandon was here, trying to get in and finish what he had started.

  She stood up hesitantly, her head pounding as if she had a hangover, though she’d had nothing to drink. On the tiny table, the claw hammer lay where she’d dropped it the night before, a small dark something crusted on the round end. Was that blood?

  She looked through the peephole and saw a middle-aged woman she didn’t recognize and two uniformed police officers. A sudden horrible thought occurred to her: Had she killed Brandon? He’d shouted after her as she ran down the hall, but as far she knew, he hadn’t pursued her. Had she left him there with a brain hemorrhage, dying on the floor without help? Was that why the police were here?

  “Naomi Sumner?” the woman called through the door. “This is metro PD. We need to ask you a few questions.”

  Naomi took a deep breath and opened the door.

  “Are you Naomi Sumner?” The woman reminded Naomi of her seventh grade art teacher, gaunt and with a perpetually stern expression on her face. When Naomi nodded, she said, “I’m Detective magda Schneider. May we come in?”

  Naomi thought of the hammer on her table and shook her head. “This isn’t a good time.”

  “You can let us in, or you can come downtown and chat in an interrogation room,” Schneider said. “It’s up to you. I thought here would be more comfortable.”

  “What is this about?”

  “We’re investigating the death of your coworker, Min-seo Cho. We just have a few questions.”

  “I already answered them. The police came by the office just after she died.”

  “I’m afraid we’ve uncovered some more evidence since then. We’re going to need a sample of your DNA. If you’re innocent, you should have nothing to hide.”

  “I thought you just had a few questions.”

  “Ms. Sumner, it would be easier for all of us if you came willingly. You don’t want to leave here in handcuffs, do you? Or spend the night in jail?”

  As she said it, one of the uniformed cops unclipped his handcuffs from his belt, and the other rested his hand on his sidearm. Naomi was pretty sure those were idle threats—if they could have arrested her, they would have done it already without all the small talk—but even so, it was pretty intimidating. She found herself wanting to agree, wanting to go with them.

  “I don’t think so,” she said. She started to close the door.

  “Hey, maggie, look at her sleeve,” one of the cops said.

  Naomi looked at her own sleeve just as the detective did and saw the blood there. She was still wearing the same sweatshirt she had worn the night before; she had curled into bed without changing it, unaware of the blood. It had to be Brandon’s.

  They moved toward her, and she tried to push the door closed on them. Schneider put her foot in, blocking it, and Naomi turned and ran back into the apartment. She snatched the hammer off the table with the desperate idea of hiding it, though where she could have hid it in her bare, closet-sized apartment she had no idea. A cop pushed through the door, and when he saw the hammer, raised his pistol and shouted for her to drop it.

  She let go of it immediately, terrified, adrenaline streaking through her body likes bolts of lightning. The cop grabbed her shoulders as the other two pushed through the door. He threw her facedown on her own bed and forced her arms behind her back, roughly cuffing them and holding her down. To her left, she saw Detective Schneider taking pictures of the fallen hammer.

  “Looks like he was telling the truth,” one of the cops said.

  “I wouldn’t go that far,” Schneider said. “When a woman attacks a man with a weapon, there’s almost always more to the story.”

  “He attacked me,” Naomi said. “He tried to rape me.”

  Schneider gave her a calculating look. “And Min-seo? What happened to her?”

  Naomi returned the gaze as best she could from her cuffed position on the bed. “I don’t know.”

  “We’ll see about that.” Schneider looked disappointed. “Despite what you may hear about the police department, I really am interested in the truth.”

  They left her in a featureless interview room for what felt like hours, presumably to wear her down before they started questioning her. She welcomed the time to think. The question was, what to tell them? She had no compunction about describing the confrontation with Brandon the night before. She had used the hammer in self-defense, and if he had told them otherwise, she would be glad to set the record straight. She couldn’t control whether they would believe her, but there was no reason to hide.

  With Min-seo’s death, however, there was plenty to hide. She didn’t care about protecting the company, exactly, but she did care about protecting the Mikes. If the company went out of business, or its assets were seized, the Mikes’ world would be shut down, and they would all be killed. She had to prevent that. Which meant lying to the police. But how could she lie convincingly, when she didn’t know what evidence they had?

  Obviously something new had turned up. Previously, the only tie to Naomi and Brandon had been that Min-seo was an employee of their company. She hadn’t been killed on company property, or while on the job, and so there was no reason to suspect they had anything to do with it. Now they had new evidence. And where did they get that? Brandon had apparently told them she had attacked him, which meant he had probably told them she had killed Min-seo, too. He might have given them the car, with her fingerprints and hair on it. It was his way of taking revenge.

  But could he really have been so stupid? How did he expect to run the company if she was arrested? Probably he thought he could hire someone else like Min-seo, or ten such people, and they could do her job. He didn’t know about the Mikes, or how Min-seo had culled them, or why she’d ended up dead. He thought it was some malfunction or code error. He had no idea.

  Finally, Detective Schneider came into the room and shut the door behind her. She sat in the chair across the table from Naomi, her face as thin and stern as it had been in her apartment. “You nee
d to start telling us the truth,” she said.

  The time to think had convinced Naomi that there was nothing to be gained by talking. The truth was not believable, and any lie would only harm her when it was found out. She needed time to come up with a strategy, and she probably needed a lawyer. So Naomi did what came most naturally: she stayed quiet.

  Schneider urged her to talk, told her she couldn’t help her if she didn’t tell her side of the story. She told her that silence made her look guilty, and if she had nothing to hide, she might as well set the record straight. Finally, she slammed her palms on the table and leaned into Naomi’s face, threatening the worst that prison had to offer. Naomi didn’t answer.

  Schneider sat, apparently defeated, and gave her a compassionate look. “I’ve got your fingerprints on the car that killed Min-seo Cho. You’re sure you don’t want to tell us how they got there? Because I’m pretty sure a jury’s going to assume it was from running her down and then backing over her again to make sure she was really dead.”

  Naomi shook her head.

  “And what about your boss? Want to tell me what happened there?”

  She answered softly. “He attacked me.”

  “I can see the bruising on your throat,” she said. “Did he rape you?”

  “He hurt me,” Naomi said. “He might have done more.”

  “But you hit him with the hammer before he could.”

  Naomi fell silent again. She thought Schneider might really be sympathetic, at least with this part of the story, but there was nothing to be gained by making a confession.

  “Was Mr. Kincannon in the car with you, when Ms. Cho died? Was he at the wheel? If you’re afraid of him, we can protect you.”