Three Laws Lethal

by David Walton

  CHAPTER 18

  At first, Naomi had enjoyed going to work in the dingy little office on the fiftieth floor. Half the time, she was the only one there, she had access to a powerful computing cluster, and the Mikes were doing everything she wanted them to and more. Now, however, her days were filled with her constant and increasingly difficult campaign to keep Min-seo from discovering the Mikes’ world. Her unfamiliarity with the code baseline had bought Naomi a little time, but Min-seo was every bit as smart as her résumé promised, and she wouldn’t be fooled for long. Naomi’s vague responses about the core of the fleet scheduling algorithms weren’t going to satisfy her forever.

  So far, Naomi had told her it was a proprietary deep learning algorithm, the secret of which was too competition-sensitive to share. If Brandon heard her saying that, however, he would force her to tell, since the whole reason he’d hired Min-seo was to teach her the critical parts of the code that only Naomi

  19 0

  understood. Even Brandon didn’t know about the Mikes’ world and how the AI really worked. He thought she had just written an especially effective but compute-intensive version of the same basic concepts that were in use in algorithms around the world.

  The Mikes continued to deliver cars to their customers with creepy prescience. Even Naomi, who knew how it was being done, was disturbed by how often she walked out of a building to find a Black Knight car pulling up to the curb. She herself hadn’t decided how long she was going to stay, and yet there was the car, waiting for her.

  The customers seemed to appreciate this, at least judging by the way sales increased. Brandon kept expanding their fleet and hiring new people to manage different aspects of the growing business that he could no longer manage himself. He interviewed and hired business developers to bring the concept to other cities, and soon Black Knight cars were transporting customers in Philadelphia, Baltimore, and Washington DC, as well as New York. He popularized the slogan “Where you need it, when you need it.” They were mentioned in national news stories at least once a week.

  Like any deep learning system, there was no way to describe how the Mikes anticipated the movements of their customers so well. There was no log to dump of possibilities considered or deductions made. Instead, all the metrics of the city’s life and movements formed a mesh of numbers, with certain numerical scores chosen because they rose to the top. The Mikes didn’t know why they did it. They didn’t know the why of anything. They just acted on the options that gave them the best chance at passing on their genes. It was the same behavior that allowed them to exploit their simulated world with such success.

  Naomi kept late hours, but she encouraged Min-seo to go home early. “We’re not paying you for overtime,” she would say. “Go home, eat dinner, see a show. Life isn’t all about work. I’ll be leaving soon, too.” Of course, she never did. Midnight often came and went with Naomi still in the office alone, studying the Mikes and their world, watching for new behaviors and imagining new ways to challenge them. She could do the same thing from her apartment, of course, but her apartment wasn’t any more comfortable or inviting. The empty room was like her secret sanctuary in the library had been, once Brandon and Min-seo left for the day. In fact, since it had a lock on the door, it felt even safer.

  Which was why she was shocked when, one Friday night after ten o’clock, a key turned in the lock, the office door swung open, and Min-seo walked in.

  “I thought I might find you here,” she said.

  She wore an evening dress, black and flattering, with heels and a thin black purse.

  “What are you doing?” Naomi asked.

  “I might ask you the same. It’s ten at night.”

  “I’m working,” Naomi said. She wanted to curl up and hide. This was her place. This woman didn’t belong here.

  “If you have that much work to do, you should let me help,” Min-seo said. “I’d try to convince you how capable I am, but that’s not the problem, is it?”

  “I don’t know what you’re talking about.”

  “I’ve been there,” Min-seo said. “I found the simulated world you’ve been hiding. I saw the people who live there, and what you’re making them do.”

  Naomi felt a hot flush of anger and fear spread through her body. “You broke into my account?” Min-seo tossed her hair. “No, of course not. Mr. Kincannon gave me access. I told him you wouldn’t give me the access I needed, so he called the cloud provider, created a new account, and gave me full admin.”

  “He gave you . . .” Naomi choked on the words before she could say them. Full administrator rights meant Min-seo could do anything. She had the same power Naomi herself did.

  “Don’t worry, I won’t abuse it,” Min-seo said. “‘With great power’ and all that. And it’s brilliant. I can’t believe you were able to create such a world. You could have published several times over.”

  “It’s not about that.”

  “You didn’t create this world for driving, did you? That just came after.” There was awe in her voice. “What you’ve got here, it could do anything. You’ve got a general problem-solving machine. Semi-intelligent algorithms that can learn to do any job, with the right incentive.”

  “You can’t tell anyone about this.”

  “I understand. This is the biggest proprietary idea ever invented.”

  “Not just that. You can’t tell other employees. You can’t tell Brandon. If other people find out, there’s no telling what they’ll do. They’ll enslave them. Hurt them.”

  Min-seo paused. She took a step forward, her heels clicking on the floor, and cocked her head. “You care about them,” she said.

  Naomi didn’t answer.

  “The creatures in the simulation. You think they’re real.”

  Naomi shrugged, helpless. “I don’t know,” she confessed. “I tell myself they don’t really think or feel anything, but the arguments all sound hollow to me. I can’t make any distinction between us and them that seems to matter.”

  “Wow,” Min-seo said. “Just . . . wow. I have to think about this.” She pulled her chair out from behind her desk and sat, elegant in her dress. “Can you talk to them?”

  “No. And they don’t talk to each other, either, not in anything I recognize as language. I don’t think that’s required, though. If they’re sapient, that’s something internal—an awareness of their own distinct existence. An inner thought life. The only ones who could know for sure if they’re sapient are the Mikes themselves.”

  “I’m sorry, the what? The Mikes?”

  Naomi blushed. “That’s what I call them.”

  “Named after somebody you know?”

  “No . . . it’s from an AI in a science fiction book.”

  “Wait, I know that one! It’s Stranger in a Strange Land, right?”

  “The Moon Is a Harsh Mistress. But yeah, Heinlein.” Min-seo shook her head, embarrassed by the mistake. “That’s what I meant.”

  “Close enough for the nerd club,” Naomi said, smiling. Maybe this would be okay. Maybe it could work. “There are some problems,” she said. “The biggest one is that they’re growing. Their world is expanding, and I can’t control it, not without undermining the whole mechanism by which they do their job. But the amount of computing power is expensive, and growing fast. At some point, it’ll grow faster than the company’s profits, and then we’ll be in trouble.”

  “I can help you.”

  Naomi sighed. “You’ll follow my lead? You’ll do everything I tell you, and nothing without permission?”

  Min-seo shrugged. “You’re the boss. Well, technically Brandon’s the boss, but . . . he really doesn’t know?”

  “He doesn’t know.”

  “Wow. And you don’t think we should tell him?”

  “No!” Naomi looked away, not meeting the other woman’s eyes. “He can’t know. He wouldn’t understand, and he might . . . he just can’t, okay?”

  Naomi’s heart was beating hard. This was a bad idea. How co
uld she trust Min-seo? Other people were unpredictable. One moment, they seemed to think like her, and then the next, they said something to make her realize they weren’t on the same page at all. Other humans could be so . . . alien. And trying to make them understand by speaking words just never worked very well. It was better when she was on her own.

  She took a deep breath. There was no getting around this woman. She had to work with her. She had worked with people before. She could do this.

  During the days, at least while Brandon was around, Naomi and Min-seo continued as usual, improving the suite of software by which customers called for cars, giving them more options and finding ways to make the experience more convenient. They started integrating other companies’ apps into their own, so that they could report, for instance, what Uber or Lyft cars were available as well as what Black Knight could provide. The vast majority of the time, a Black Knight car could get there faster, and provide transportation for less, so there was little chance of losing business. Instead, the new feature solidified their customer base by making it clear that no other transportation service could compete.

  At night, or when Brandon was out, they studied the Mikes’ world together, adjusting the configuration, improving the quality of life for those able to pay for it, further stratifying their society to encourage them to drive cars and drive them well. They watched for signs of sapience, and argued about what counted as programmed activity and what as emergent behavior. Most of all, however, they argued about killing them.

  “It’s just not sustainable,” Min-seo said. “The world is ludicrously big already, and it’s getting worse exponentially. You’re going to drive this company into the ground.”

  “There’s got to be another way,” Naomi said, knowing she had said it before. “We can’t just murder them for expediency.”

  “They die all the time! There are millions of them, dying as we speak. They barely live long enough for us to have this argument. We just need some of them to die a little quicker.”

  The city was dark outside the windows, as dark as it ever got, the glare of electric lights taking the place of the sun. As usual on a Friday night, Min-seo had left to get dinner and then returned, dressed to the nines, to spend a few hours working before going out clubbing. She wore a black mini-dress with a tiny jacket and heels, which made Naomi feel frumpy and defensive in her jeans and sweatshirt.

  “We can reduce the birthrate,” Naomi said. “Slow their growth.”

  “You know that doesn’t work. The only reason they do what they do is because we reward them with reproduction. Reducing the birthrate just reduces the efficacy of the algorithm without solving the problem. We need to cap the size at something reasonable, not allow them to grow indefinitely. We do that by culling them back whenever they get too large.”

  “Won’t that cause them to learn to slow their own birth-rate?” Naomi said. “To avoid the inevitable holocaust?”

  “Not if we just do it at regular intervals. That way nothing they do can change it. It’s like pruning a tree. We keep it manageable.”

  Naomi didn’t answer. She hated this, and she hated that Min-seo was involved. Maybe the best option was to take her simulation and leave the company. That way nobody could tell her what to do. If she didn’t need them to drive, she could reduce the birthrate, even let them live longer. Though she knew that was impractical. She couldn’t afford to pay for even a day of computing costs at this point. Pulling out of the company would cause an even worse genocide than staying in.

  “They’re not human,” Min-seo said. She stood, looking down across the desk at Naomi, imperious in her heels. “Don’t get emotionally attached.”

  “That’s what people with power have been saying for millennia,” Naomi snapped. “Those inconvenient refugees aren’t really people, so we can leave them to die. Those slaves aren’t really people, so we don’t have to disrupt our economy to give them rights. Those women—”

  “Don’t lecture me about human rights. I don’t need to take that from you. The difference is these simulations aren’t human. The only humanity they have is what you’re projecting onto them in your head. Snap out of it, Naomi. This isn’t a science fiction story. It doesn’t have to be so difficult.”

  “I can take care of it,” Naomi said.

  “I know you can. I saw that script you wrote. You know what needs to be done as well as I do. You’ve known for months.”

  Naomi glared at her. “Go away,” she said, hating that her voice came out pleading. “Go out bar-hopping or man-chasing or whatever you do. You have no business reading my code.”

  Min-seo sighed. “Actually,” she said, “I do. That’s why Brandon gave me admin rights. He doesn’t want you keeping secrets.”

  “This is my software,” Naomi said. She felt the flush of anger in her cheeks, mixed as it always was with embarrassment at making a scene. “It’s my simulation. It was mine before he even started this company. He may have given you admin, but that doesn’t mean you have to use it to sneak around behind my back.”

  Min-seo smiled, but it was a pitying smile. A superior smile. “I’m afraid Brandon wasn’t exactly honest with you,” she said. “Before I even met you, he told me specifically to sneak around behind your back and read your code. It’s why he hired me. He didn’t trust your explanations for why you needed so much computing bandwidth, and he wanted me to get to the bottom of it. I’m sorry.”

  Naomi slowly stood from her seat, bringing her eye to eye with Min-seo, or as close as she could come. “You didn’t tell him, did you? About the Mikes?”

  “Not yet. But I will. I have to, Naomi. He’s paying for them. It’s the company’s biggest expense, and it’s going to destroy his business if we can’t keep it in check. He’s got to know the truth.”

  It occurred to Naomi that this was just the sort of situation that got people murdered in the movies. The well-meaning-but-naive female confronts the villain with a secret he’s worked hard to conceal. She threatens to tell the world. She does it alone, of course, and at night, without telling anyone else where she’s going or sharing the secret with anyone. Min-seo was practically begging to get killed. She was lucky Naomi wasn’t a movie villain, or she might have found herself garroted with an ethernet cable.

  Could you garrote someone with an ethernet cable? Naomi didn’t know. It was a lot thicker than a piano wire, of course, but it would probably be effective enough for strangulation, anyway. Naomi had no letter openers to hand, nor any large paperweights to bludgeon with. The windows were too small to push anyone out of. Then she remembered the claw hammer still sitting on top of the filing cabinet by Brandon’s desk. He had used it to assemble the office furniture, and had never moved it. She actually took a step back toward it before she realized what she was doing. What was she thinking? She wasn’t going to kill Min-seo, any more than she was going to kill any of the Mikes.

  “I should never have written that script,” she said. “We need to leave the Mikes alone, to grow and develop on their own. They’re people. It doesn’t matter that they’re not human. It doesn’t matter what’s easier for us. If it costs money, then that’s what it costs. Do you hear me? We can never run that script.”

  “Actually,” Min-seo said, “I already ran it.”

  Naomi felt a chill stopping her heart, making it hard to breathe. A black rage filled her, and she glanced again at the claw hammer. “You had no right.”

  “This isn’t your personal little playground,” Min-seo said. “It’s a business. To make money. I’m sorry, but it had to be done. You know it did. It’s just a simulation, no more or less real than any number of realistic video games.”

  “And how could you possibly know that?” Naomi said, in a whisper so soft she could barely hear it herself. She had no stomach for conflict; she never had. She wanted to run home to her apartment, lock the door, and curl up in a ball on her bed.

  “I’m sorry it had to be like this,” Min-seo said. “If you had actually told
your boss what he was really paying for, he never would have had to hire me in the first place. Of course, he would have told you to run the script.”

  “You had no right,” Naomi said again, angrily blinking away tears from the corners of her eyes.

  “It’s done,” Min-seo said. “I suggest you resign yourself to the fact, because it won’t be the last time.” She turned, swinging her long dark hair behind her, and headed for the door. “The nightlife calls,” she said. “I’ll see you on monday.”

  The script did exactly what Naomi had designed it to do. Of course it did. It hadn’t been complicated to write. When Min-seo executed it, millions of Mikes suddenly died, terminated before they could replicate. There was no sign that the change made any difference to the Mikes that remained. They continued to do their jobs, competing to earn enough light to survive and pass their genes on to the next generation, the same as they had always done. They didn’t get angry. They didn’t grieve.

  Had she been wrong? Min-seo’s arguments, after all, were the same ones Naomi had been telling herself for months. She had never really considered the Mikes sapient until the time had come to kill them. Had she let sentimentality cloud her judgment? Or was it just that Min-seo and Brandon had forced her hand? She had never liked other people telling her what to do, especially not where her software was concerned. Maybe she owed Min-seo an apology. And Brandon too, for that matter. Maybe all her science fiction books had made her imagine things that didn’t exist. When it came down to it, it was pretty arrogant for her to think that she would be able to create truly intelligent machines. She was no god. She was just a self-absorbed, antisocial nerd with nothing better to do on a Friday night than work in a dark office imagining impossible wonders. Fabricating fairy-tale companions like a child with an imaginary friend.

  An hour and a half after Min-seo left the office, Naomi shut down her machine and walked outside to find a Black Knight car waiting for her. How could it possibly have anticipated her departure so well? She often stayed later than this. She wasn’t special; the amount of stored light credited to the Mike driving the car was the same as for any other customer. It wouldn’t have been worth it for the car to linger here in the financial district, not on a Friday night. There would be a lot more fares on the Lower east Side, or across the Williamsburg Bridge in Brooklyn. And yet here it was.