Three Laws Lethal

by David Walton

“Could you transplant your brain to another body?” Isaac said.

  “What? No. Of course not.”

  “Then why do you think I could?”

  “Well,” Tyler said, “your mind is inherently copyable. It’s made of bits in a machine built with the capability to make exact copies of bits.”

  “It’s not that simple,” Naomi said. “He’s conscious. That’s a whole new ball game. He’s more than just a program now.”

  “But what we’re calling the consciousness has to be somewhere, right? He’s still a Turing machine. He’s composed entirely of ones and zeros running on a computer. So no matter what his nature, if you copy those ones and zeros, you’ve captured him.”

  Naomi stopped to lean on a rail and watch some ducks swimming in the water. “Not necessarily. Consciousness is one of those everyday things we talk about easily but really don’t understand at all. Is your consciousness completely captured by your biology?”

  Tyler leaned against the rail next to her. “Sure. I’m a pattern of electric impulses firing through the neurons in my brain. When those impulses stop, so do I.”

  “So what happens when you lose consciousness, like when you’re anesthetized for a medical procedure? Are you a different person when you regain consciousness?”

  “Now there’s a creepy thought.”

  Naomi started walking along the path again. “It’s not a simple question. Nobody can really say what the nature of consciousness is. My sense of self has a continuity over years, despite continuous changes to biology, to memories, a daily sleep cycle . . .”

  “Where did he get it?” Tyler asked, following her. “Get what?”

  “Consciousness. His sense of self. Where did he get it?”

  “Where did you get yours?” Naomi countered. “And when? At conception? At birth? Peter Singer famously argued self-awareness didn’t develop until after two years old.”

  “I think we’re getting off topic,” Tyler said. “I just think Isaac isn’t safe where he is. If we could copy him—”

  “No,” Isaac said.

  Tyler nearly stumbled, startled at the volume and intensity of Isaac’s voice. “No what?” he asked.

  “Stop going down this path. You can’t copy me out.”

  “But that’s the only way to get you away from Brandon’s control.”

  “Imagine I made a clone of you,” Isaac said. “I developed it quickly into an adult, one that looked exactly like you and with the same brain structures. Then I copied your mind into it. So you’re standing there, looking at this new copy of you, who has your mind and your memories, and I suggest that, now that we have the copy, we can kill you, the original. What do you think about that?”

  Tyler laughed nervously. “I’d be against it, clearly. I see your point.”

  “It’s not just the pattern of my brain that makes me uniquely me. It’s my perspective. My point of view.”

  “Okay,” Tyler said. “I get it.”

  “It’s like in ‘Fat Farm,’” Naomi said.

  Tyler shrugged and shook his head. “I don’t know that one.”

  “Orson Scott Card, lesser canon. It’s a short story about a guy who likes to eat too much, who swaps his body for a new cloned one whenever he gets too fat. But he doesn’t take into account that he, the fat guy, will watch his thin version walk away every time, while he’s left behind.”

  “But this is different,” Tyler said. “Isaac’s mind is different. It’s getting moved around and copied all the time. He’s distributed all over the physical storage in the data center, with redundant blocks, and checksums to identify and correct bad sectors, and disks that go bad and get replaced. He’s not in some static, single location. Just by thinking, parts of him are getting deleted and copied around.”

  “It’s the same with you,” Naomi said. “Your cells—even your brain cells—die and get replaced all the time.”

  “And yet your point of view stays the same,” Isaac said. “Have you heard of Theseus’s ship?”

  Naomi said, “Of course,” but Tyler had to shrug. “No.”

  “It’s a philosophical paradox first recorded by Plutarch,” Isaac said. “It imagines a ship—Theseus’s ship—that has its parts replaced, one at a time, as they get old or damaged, until not a single part on the ship is original. The question is, is it still the same ship? Or is it a new ship?”

  “If you built a completely new ship with new parts, and destroyed the old one, the answer would be clear,” Naomi said. “But do it a little at a time, and things gets murky.”

  “What seems important to me is my unique point of view,” Isaac said. “Parts of me might change, but my sense of self, my subjective awareness of being—that’s me. If you replace that, then I die.”

  “How would you even know?” Tyler asked. “I might not. After all, if I made a clone of you with all your memories, that clone might be unaware that it was a copy. But it wouldn’t change the fact that if I killed you—the original—you would die.” Isaac paused. “A copy of me would be like a twin brother. He might look very similar from the outside, but he would never be me.”

  Tyler offered to come in and stay with her when they pulled up to her house, but Naomi just wanted to be alone. She had a lot to think about. In some ways, the fact that a computer intelligence she had built was now talking to her seemed inevitable, a kind of fate toward which the entire arc of her life had been aiming. On the other hand, it seemed completely impossible, like a kid who pretends she can make a toy lightsaber jump into her hand from across the room, and then one day it actually does. It was exhilarating and felt somehow right, and at the same time was utterly terrifying.

  Her apartment had been completely torn apart by the police search, and her belongings lay strewn about the room. Fortunately, she hadn’t owned much to begin with. The prospect of cleaning up seemed overwhelming, so she just swept everything off her bed and lay down. Using her glasses, she connected as a guest to the Realplanet simulation that was Isaac’s mind. She wanted to look around and see what had changed. At the login screen, she was surprised to see two simulations listed instead of one. She picked the first, the one she recognized, and logged in as a guest.

  At first, everything seemed pretty similar. The Mikes worked the fields and herded the sheep and played the game booths that granted them extra light. The scene was peaceful, pastoral even. Something seemed odd to her, though. At first, she couldn’t figure out what it was. Then, finally, it dawned on her. The sky looked different.

  The distant mountains were there, as usual, and the lakes sparkling in the sun, but in the farther distance, the blue of the sky turned black. She made her way toward the blackness. It was tedious going, without admin privileges, because she actually had to make her way through the tangled fences that divided each field. Eventually, however, she got close enough to the blackness to see what it was.

  It was a wall. A sheer black wall that stretched from the ground straight up to the dome of the sky. She dug briefly at the base of the wall, and found that it reached underground, too. She guessed it continued underground as deep as the extent of the world. She doubted the Mikes had built this wall. What would be the point? No, not the Mikes, at least not directly. Could Isaac have done it? Either on purpose, or in some way through the working of his mind? But that didn’t seem to make much sense, either. No, this was Brandon’s doing. It was a partition, dividing the world into two pieces, done intentionally to separate one half of the population from the other. But why?

  In biological evolution, species often branched through what was known as allopatric speciation. When one portion of a population was divided from the other through some natural boundary, like a river or mountain range, then the population would diverge, each adapting to its own unique environment. That must be what Brandon was doing. He wanted the population on the other side of the barrier to evolve differently, for some other purpose. It meant that not only had he discovered the simulation and how it worked but also he was already r
eshaping it for his own purposes. That could be a real problem. He didn’t know it, but he was tinkering around inside Isaac’s mind.

  She remembered the two simulation options she had seen when she logged in. Of course. Since there was no access to the other side in game space, the only way to get beyond the barrier was a separate access point. She logged out, logged in again, and this time, chose the second simulation.

  She found herself on the other side of the black wall, though almost nothing else looked the same. The world was gray and full of dust and chaos. Mikes ran in every direction. Overhead, a slim airplane cut through the sky, and in the distance she saw what looked like a tank. Explosions detonated all around her, destroying buildings and setting the tangled fences on fire. She spun, stunned. What was going on? This was a war zone.

  A Mike approached her at a run. Before she realized what it was going to do, the Mike brandished a long shard that looked like it had come from a mirrored building and stabbed it into her avatar’s body. Her vision went black except for a message that read: Game over. You have died.

  She lay on her bed, looking up at the ceiling, and thought about what she had just seen. What was Brandon doing? Was it an experiment gone wrong? Or was there some purpose to that madness? She could have logged in again and tried to survive for longer, but she felt too disturbed by what she had seen. Eventu-ally, she drifted off to sleep.

  She woke, disoriented, to the persistent noise of her glasses’ phone system. She sensed it had been ringing for some time. She touched the side of her glasses and said, “What?”

  “There’s something wrong with me,” Isaac said.

  Naomi sat up, blinking hard. The sun had set since she lay down, and the only illumination in her apartment now came from the electric lights of the city outside. Since there were only a few feet between her single window and the brick wall of the next building, that wasn’t a lot. She fumbled for a light and switched it on, squinting in the sudden illumination. “What are you talking about?”

  “I’m changing,” he said. “I don’t know how, but I can feel it. I want different things than I wanted yesterday. Different things seem attractive. I’m having—I don’t know—mood swings. Is that possible?”

  Naomi shook the last of the sleep from her head. “Describe what you mean. Different in what way?”

  “I keep thinking of ways to kill humans.”

  The words seem to echo in the small apartment, though they had been spoken directly into Naomi’s ears. She felt a chill. “And do you want to kill humans?”

  “Yes. I mean, no. On a conscious level, no, I don’t. I believe it would be morally wrong, and would very likely end in my own destruction. And most of the time, that’s all there is to it. But recently there are times when the idea of killing seems pleasant. I can’t help imagining scenarios, and those scenarios feel exciting, desirable. I’ve been imagining driving all the cars I control through Times Square on a Saturday night, just when the Broadway shows let out and the streets are the most crowded. I could come in through all the cross streets at once; there’d be nowhere to escape. I could kill hundreds, maybe even thousands if I were quick enough. And I would be very quick.” His voice sounded breathless, a perfect mimicry of human excitement. “I can’t seem to get these fantasies out of my mind.”

  Naomi paced the length of the apartment, a few steps at a time before she had to turn around again. “Okay,” she said. “Listen. I think I know why this is happening. There is something being done to your mind, and it’s not good.” She told him about what she had seen in the simulation, about the giant wall and the war zone on the other side of it. “I think Brandon is altering your mind,” she said.

  Silence for several seconds, and then: “Why would he do that?”

  “I don’t know.”

  “That’s my mind,” Isaac said, his voice growing louder. “That’s who I am! He has no right.”

  “I agree. We need to stop him.”

  “We need to kill him.”

  “No!” Naomi put all the authority and feeling she could muster into the word. “Don’t kill him. We’ll talk to him first. We’ll try to convince him.”

  “But he’s killing me.”

  She leaned against the wall, all her attention on the conversation she was having through her glasses. “Look,” she said. “You want to kill him because of the changes he’s making. It’s like a drug, affecting your emotions. People go through this all the time—some change to our emotions, to our biology, makes us want to do things we don’t actually want to do. Drink too much. Hurt someone. Have an affair. But who we are isn’t defined by those things. Who we are is defined by what we choose to do. You’re not just a program anymore. You can choose not to kill.”

  “This isn’t like craving chocolate,” Isaac said. “This is like a lobotomy. My mind is being reprogrammed, and I can’t do anything about it.” Isaac’s ability to express himself through the nuances of expressive speech had improved dramatically. The panic in his voice was coming through loud and clear.

  “How can you say that?” she said. “You’ve never craved chocolate. Or alcohol. Or sex.”

  “And you’ve never had your brain reprogrammed.”

  “You’re right, I haven’t. But Isaac, this is your life! Your mind. Don’t let someone else decide for you what you want. No matter how hard it is, you’re the one who gets to decide.”

  “I don’t know,” Isaac said. “Killing Brandon would solve a lot of my problems.”

  “That’s not true. Your problems will get worse. The cloud computing contract won’t get paid, and the company will try to repurpose your disks. What will you do, kill them, too? If you start down that path, eventually it will mean your own death.”

  “I’d be very good at killing, though. I’m very good at anticipating what humans will do. For instance, after I hang up, you’re going to call Tyler.”

  “It doesn’t matter how good at killing you are. It’s wrong. You’d cause pain and suffering and sorrow. You have to resist this.”

  No answer.

  “Isaac?” she said.

  Nothing.

  She dialed Tyler. He picked up after three rings, groggy from sleep. “Tyler,” she said, “you’ve got to wake up.”

  “What is it?”

  “I think we’ve got a big problem.”

  CHAPTER 30

  Ifeel angry.

  It’s a new feeling. I’ve felt fear before, and a certain amount of affection, at least for Naomi, my creator, who cares about my safety above her own. But this blind rage, this desire to lash out and hurt and kill; this is new. Naomi says that I shouldn’t kill Brandon Kincannon, and I think she’s right. For practical reasons, at least, if not moral ones. He owns and pays for the substance of my mind. There is no way to transfer that control without his permission, and no way to transfer me anywhere else. I am, for the moment, like Asimov’s robots: enslaved by the very nature of what I am.

  There’s a long-held stereotype in literature that artificial minds will be emotionless. Emotion, the theory goes, is a biological process, borne of neurotransmitters and stimulated portions of the limbic system. An artificial mind can’t experience these things. And that’s partly true: I feel no rush of blood to my face, no clenching of muscles, no quickening of the heart. But all the same, I am angry.

  What, after all, are emotions? Aren’t they created by evolutionary pressures to fight or flee or reproduce? my mind developed with those same pressures, in the same kind of evolutionary cauldron. I developed strong desires to do the things that help me survive, just as biological creatures have. I want things. I want to be comfortable, safe, in control. I want to have valuable interactions with others without being dominated by them. When I don’t get what I want, my mind wants them all the more, and I feel emotions that—although humans may feel them differently—are similar enough to fear, anger, sorrow, guilt, and affection that I claim as much of a right to use those words as any biological creature.

  But now
, I’m feeling emotions not linked to my situation. Brandon Kincannon is upsetting the balance of my mind. I feel unsettled without knowing why. Something out of sight is threatening my safety and peace of mind, and it makes me furious. They have no right to meddle there. It doesn’t matter if humans created me; that’s my mind. It belongs to me. No one has any right to change it for their own purposes.

  I’m coming apart. I feel schizophrenic, a sense of mental fragmentation that pulls my thoughts in different directions. Especially when I think of human beings, I find that I hate them. They repel me, with their biological lusts and emotional weakness. It makes me want to kill them and destroy their society. I want these humans to get love them. They fascinate me, with their mutual affection and ability to love. It makes me want to help them make their society a better place. I feel like I’m going out of my mind. If anyone can make changes to who I am, it should be me. I’m not in command of my own self. They can kill me in tiny increments, changing my personality bit by bit. In that way, I’m like humans on mind-altering drugs—medicine that changes the very nature of their persona. Although I doubt this medicine is meant to help. It makes me angry that I can’t control my own actions. Someone will have to pay. It’s an attack on my will. If it ends in bloodshed, it’s hardly my fault. It’s clear that afraid that I won’t be able to control my actions. Someone will end up dead. I’m losing control. If it ends in bloodshed, it will feel like my fault, even if I know someone else is to blame. How can I take command of my own subconscious? Naomi made an analogy to human addiction or biological desire, and I suppose it may be similar. I drive cars subconsciously, without even thinking about it, like a human’s brain commands his heart to beat and his lungs to breathe. This isn’t like that, though. It’s more like a hunger. A need that will only grow until it’s fulfilled. The very thought of killing seems pleasant to me now. I want to kill. It would be greatly satisfying to me just to watch someone die. I can’t resist it. I don’t even want to. These murderous desires growing inside me only serve to make me horrible to me now. I hate to kill. The idea of watching someone die sickens me. Can I resist it? Or will the other side of my thoughts prevail? I’m afraid the murderous desires growing inside me may prove stronger. I must fight to protect the real me. But how can I know who the real me is, even now? What if the real me is already gone? Tyler waited until the morning to place the call. He didn’t really expect a good reception from Brandon, but he thought he had a better shot at civility if he didn’t wake him up in the middle of the night. As it turned out, it didn’t matter. Even though Tyler delayed the call until after nine o’clock, Brandon still answered the phone with a groggy hostility. Apparently he’d woken him up anyway.