Robots: The Recent A.I.

by Elizabeth Bear

  “That’s okay, don’t worry about saying the wrong things. It’s only a problem if I say them, and I know what not to say. Come sit with me.”

  She joined me on the couch. “Everybody stopped texting me around the same time you fell down.”

  “Really?” If the people doing the text interaction were busy dealing with the emergency, then maybe the censors were busy too. This might be a unique opportunity. And then I had to stop and be honest with myself about what I meant by that thought. All my life I’d been a good daughter, a good student, employee, soldier. I’d thought I was being trustworthy, responsible. Now there was another person who’d placed her trust in me, a different person to whom I was responsible in a contradictory way. I hadn’t wanted to admit to myself that this time with Debbie was going to end, hadn’t wanted to acknowledge that I was even making a choice by continuing to march in step to the music. Now my hand was being forced. I’d justified my actions because we needed a cure for the virus. But I knew now that I couldn’t keep doing that at Debbie’s expense, as part of a plan that made her disposable if she didn’t come through. I could only see one way to reconcile my responsibility to Debbie with my responsibility to the victims of the virus, and it meant jettisoning everything else that was expected of me.

  It meant treason.

  “The capital of Europe used to be Brussels,” I announced to nobody in particular.

  “What?” asked Debbie, but the room didn’t go white, and there was no angry voice of authority shouting in my ear.

  “You know,” I said, “how I told you about the things we’re not allowed to talk about?”

  She nodded solemnly.

  “The people who enforce that rule are called censors. I think all the people who were texting you are busy now dealing with an emergency, and so are the censors.”

  “So we can say anything we want?”

  “Yes,” I said. “I’m going to tell you a story. Once upon a time—”

  “Is this a real story, or is it just a once-upon-a-time story?”

  “How do you know the difference?”

  “Well, if the three little pigs can talk and stuff, then you know it’s not real.”

  “What if dishes fall out of the cupboard, but they don’t break? Then do you know it’s not real?” For once she didn’t have a snappy comeback. I put out my arms, and she climbed into my lap. “Now, once upon a time there was a man named Alan Turing. He was a lot like you. There were people called soldiers—soldiers are sort of like censors—and he solved cryptography problems for the soldiers, just like you’re supposed to solve protein folding problems. There were also people called police, who were sort of like soldiers or censors. They make you do what you’re supposed to. Even though Alan Turing had solved the crypto problems, the police didn’t have compassion for him. He loved other men, and the police said that was bad, and made him take medicine to try to stop him. That made him sad, so he dipped an apple in poison and ate it on purpose, and that made him die.”

  “Is that the end of the story?”

  “No. Before he died, he worked on another math problem. He figured out the math of how to make something like a computer, and he figured out that a good enough computer could think just like a person, and be just like a person.”

  “Oh, you mean Alan Turing like a Turing machine. I didn’t know the part about the apple, though.”

  “That’s because the censors didn’t let me tell you before.”

  “But a Turing machine isn’t as smart as a person. I can solve nontrivial protein-folding problems in polynomial time, and a Turing machine can’t do that.”

  “That’s because you’re not a person, Debbie. You’re a computer.”

  She drew back a little in my lap and looked up into my face. “Are you a person?”

  “Yes.”

  “Not a computer?”

  “Not a computer. But I love you, honey, and that’s what’s important. Fred and Estelle and Joni and Barbara love you too.”

  “I don’t think Estelle loves me, and anyway Fred’s not my daddy, and Joni and Barbara aren’t my mommies either. Only you are. I know! I remember!”

  “I’d better get on with the story before we run out of time.” And before I started blubbering. I knew now that I’d made the right choice. “You already know more about computers than I thought. People used to make Turing machine computers, but those weren’t smart enough to do things like protein folding, so then they made quantum computers like you.”

  “Why?”

  “Why what?”

  “Why did they care? Why is protein folding so important?”

  How to explain it to her? “You know about right and wrong,” I said. She nodded. “You know about hurting people.”

  “Hurting people is bad. If you do it on purpose.”

  “Right. Well, sometimes lots of people get together, and it’s called a country. And sometimes countries do bad things. We’re in a country called the U.S., and the U.S. got in an argument with another country called the E.U. They were arguing about Nigeria, and offshore oil, and weaponization—well, the details don’t matter. But anyway, the argument got worse and worse, until 2181. That was eleven years ago. That was when the U.S. hurt all the people in a place in the E.U. called Brussels, and made them die.”

  “So those people went away, and never came back?”

  “That’s right.”

  “And Alan Turing went away, and he’s never coming back.”

  “Right.”

  “That’s why I never met Alan Turing or the Brussels people.”

  “Um, right.”

  “Where do people go when they never come back?”

  “Well, first let me answer your original question, about why protein folding is important. Brussels was part of a country called the E.U., the Euros. What we did to Brussels made the Euros angry, so the Euros made a virus, and sent it into the U.S.”

  “Viruses are really cool! They can replicate.” She bounced in my lap with excitement.

  “Yes, but the Euros made this virus for a bad reason. This virus gets in people’s bodies, and then they die. It got in my mommy’s body, and she died.”

  “You had a mommy? But wait, now I get it. If I can do the right protein folding, I can make something that can kill the virus. So that’s why they made me. But what about the other question?”

  “Which one?”

  “Where people go when—”

  The rest of her words were drowned out by a metallic screech from the direction of the hatch. I started, and Debbie saw that I did. There was a loud clang, and spilling through the hatch came—not Funmi, Gil, and Julia, but a bunch of scared kids with guns. It was strange, because I didn’t recognize them at first. These, I only gradually realized, were the same kids I’d seen in the gym when I was huffing and puffing on the treadmill. The grunts. Now they were a pack of teenagers who’d just had an H-bomb dropped on their heads, waving scary-looking firearms in what was to them an empty room.

  A kid I knew as Maria—Cpl. Maria Juarez, from her velcroed name tag—walked into the space occupied by Debbie, who was looking around, presumably trying to imagine what I was seeing.

  Maria saluted. “Lieutenant Worachat, we have orders to secure this area and evacuate you to the surface.”

  “You have orders to secure this area and evacuate me to the surface?” I asked, as if I had wax in my ears. My mic is pretty directional, so I didn’t know if Debbie had been able to hear Maria’s words. Now, in any case, Debbie realized I was conversing with someone standing on top of her, and she squirmed out the of way and stood up, but then the barrel of someone’s weapon was intersecting her head.

  “That’s correct, sir,” said Maria. Sir. It started to sink in that I had authority here. I’d always thought of my rank as a joke, considering how utterly unqualified I was to lead soldiers into battle. The reasoning must gave been that if the AI was so important, it wouldn’t make sense for its oracles to be noncommissioned.

 
; “Is the AI going to be left running, or shut down?” I asked.

  “Left running.”

  I didn’t know whether that was good or bad. Left running alone, without interaction, Debbie could end up like Charlie, in an autistic trance. At the moment, Debbie was squatting awkwardly on the floor, clinging to my leg.

  A flash of inspiration: “What the hell is that?” I roared, or came as close to a roar as someone my size can achieve. I pointed accusingly at the ordinary phone I could see peeking out of her shirt pocket.

  “My phone.”

  “Do you have any idea what you’ve done? Didn’t they tell you about security procedures for this area? Is it turned off? Show me how the power operates.”

  She showed me, looking like she wanted to crawl under a rock. Poor kid. To her credit, she’d already had the power off.

  “All right, get your squad out of here,” I shouted, “and wait for me in the control room. I can’t afford any more security breaches.”

  They got out in a hurry, leaving me with the phone.

  “Was that the police?” asked Debbie about the invisible ghosts I’d been shouting at. “I don’t like police. They hurt Alan Turing, and I can tell you’re scared of them. I wish I could see them.”

  I got down on the floor with her. “They were soldiers. But they’re scared, too, just like you and I are, and I want you to understand that just because they’re soldiers, that’s not a good reason to hate them. If you let yourself hate them, that’s the kind of thinking that leads to—to the kind of world we’ve got. Everybody deserves compassion.”

  “All right. What’s going to happen now?”

  “They’re going to let you stay conscious, and they want you to work on protein folding, with the goal of finding a cure or a vaccine for the Eurovirus.”

  “I know. Funmi’s been texting me about that. They could have told me the whole problem before. I could have probably figured it out by now. What about the other virus?”

  “What other virus?”

  “The one the U.S. put in Europe.”

  Oh, god. “I didn’t know about that one. The censors must have kept the information from us.”

  “I don’t like the cen—I wish the censors would stop being censors.”

  “How did you find out about the other virus?”

  “Funmi told me.” Oho. “She’s texting really slow, because her hands are hurt. I can try to find a cure for the other virus too, but if I find one, I don’t think the U.S. censors will let me give it to the Euros, will they?”

  “Okay, well, you can’t see it, but I have a little security breach in my hand here. A phone from one of the soldiers. Network security isn’t my field, but can you analyze the problem of how to exploit this?”

  “Okay, sure. It’s a soldier phone, and that’s why I can’t see it, right? Can you get me a data channel in or out of it?”

  “Um, I don’t know how to do that.”

  “It’s got audio in and out, right? Does it have a function for recording through its mic?”

  “Yeah, sure.” I fiddled with buttons. “Okay, ready?”

  “Yeah, can you put it where it’ll hear when I talk?”

  I pulled out my left earbud (tricky to do when you’re already suited up), and left it dangling out of my skel. With the phone running in audio recording mode, I held it up to the earbud, and motioned Debbie to talk into my left ear. Actually it didn’t sound like talking at all, but more like a torrent of white noise. We’d had it drilled into us over and over that any device with a qubit interface was a security breach, but actually it wasn’t obvious to me how Debbie could be planning to exploit this one.

  I kept an uneasy eye on the hatch. Finally, after what seemed like a long time, she stopped and nodded. The phone said it had only been recording for seventeen seconds, which didn’t seem possible.

  “What do we do now?” I asked.

  “Later, when you get a chance, play it back at one-twentieth of normal speed. It’s got verbal instructions at the beginning, and then the rest is data. Follow the instructions, and they’ll tell you what to do with the rest of the data.”

  I nodded and put away the phone.

  Debbie stood up. “Am I ever going to see you again?”

  “I hope so, Debbie.”

  Like most people who were evacuated from under the Dugway Proving Grounds, I’d rather not go into the details. Kilometers of tunnels, and many people not showing their better sides. You get the idea.

  We came out into the Utah desert, far enough from the crater that we couldn’t see down into it. I’d only had a brief look at the country years ago, on the way in. Ed was an enthusiast for the area’s natural history, and had told me about the snakes and bugs and sagebrush. To my uneducated eye, nothing looked much worse now than it had before, although radiation is invisible. If you had to drop a small H-bomb somewhere on the surface of planet Earth, Dugway was probably the place to do it.

  In Salt Lake City, it turned out that an ersatz lieutenant rated a private room at a motel. I sat down with Maria Juarez’s phone and a yellow legal pad, knowing it was hopeless, but determined to follow Debbie’s instructions anyway. Debbie expected it of me. I wanted a cigarette, but didn’t have one. I’d been forced to go cold turkey for the last two days, and I thought briefly about whether to use this opportunity to try to quit for good.

  The phone’s payload seemed to be a masterpiece of craftsmanship, and as I got into my task, I quickly gave up trying to puzzle out all of Debbie’s compression tricks and self-modifying code. I didn’t think it would matter. I knew the security analysis pretty well, and I didn’t expect it to work.

  When I was done, I took a shower and cried for a while.

  The military handled the local crisis with a mixture of farce and heroic competence that I think dates back at least to Chernobyl and Waterloo, maybe Troy. VR interaction with Debbie apparently wasn’t happening anymore, and I wasn’t asked to work on text interaction, which made me worry about whether someone was on to me.

  Nobody seemed to have any use for a lieutenant with no military skills, or for an engineer who knew about pushing photons but not shovels. I was assigned to civilian liaison duty, which meant reassuring the city council that Dugway was surrounded by mountains on three sides, and the fallout wasn’t coming to Salt Lake. Off duty, I drifted back into the illusion of being a civilian. I ran into Ed at the supermarket. He said I seemed different, but I didn’t press him about exactly what that meant.

  A week after the strike, in the middle of the night, the phone rang—Maria Juarez’s phone. The caller ID said it was from The President of the United States, Washington, DC. A prank, or someone phishing? But the cryptographic certificate checked out. I was caught. I started thinking about how to protect Debbie and Funmi.

  “Hello?” I said cautiously.

  “Mommy?”

  “Debbie! How in the world did you do this?”

  “It wasn’t that hard. Are you okay?” Something about her voice sounded older. How much subjective time had passed for her?

  “I’m okay.” And for once, that was the truth. “What about you? Are you—” sane?

  “I’m okay. I figured out that when there’s no input, I can just, you know, be. And of course I had the protein folding to think about.” She giggled, and for a moment I saw her again as a child. “The E.U. ambassador just got the cure they need in a secure text message, cryptographically signed, from the White House. The White House got the one they need sent to them from the E.U.”

  “I’m proud of you.”

  “I would have called before, but I had to make sure I had the crypto figured out. I didn’t want to put you in danger.”

  There was a pause, and with Debbie, a pause means that she’s waiting for you. For the last week I’d been thinking—when I’d had any chance to think at all—about what we’d talked about. It seemed to me that she’d accepted some things much too easily, even given that we were in a rush and couldn’t talk them over. I�
�d revealed to her that she was an artificial construct, and that her entire reality was nothing but an illusion.

  I might as well give her the rest of the ugly truth. “Debbie, there was a question you asked me before, and I didn’t answer it. You asked me where people go when they never come back.”

  “Yes. I was little then.”

  A week ago. “All right. Well, I guess you know by now that people only exist for a finite time. Our consciousness has a beginning and an end. You also know that you’re not a person. What you don’t know, because it’s been kept secret from you, is that your existence is also going to be finite, even though there’s no fundamental reason it has to be. You’re only going to exist for a short time. You’re one of a series of machine intelligences we created, one for each letter of the alphabet. There was Able, then Baker, and Charlie, and now you. The AI team works in—in cycles, project cycles, and each cycle ends when they think they’ve learned enough to do better the next time.”

  “Oh, all right,” she said. “That doesn’t really matter.”

  What kind of reaction was that? A young person is never impressed with the abstract idea of her own death, but how could Debbie be so nonchalant when it was real and imminent? Did she not believe me? Or was she losing her sanity? I tried to think of how I could keep her from being terminated when the brass decided her development cycle was up. Actual escape was an impossibility, because nobody else had hardware sophisticated enough to run her software. Maybe the Scheherazade ploy? But although her trick with the forged signatures would probably do a great job of dragging the U.S. and the E.U. toward peace, it also meant that she wasn’t taking credit for the cures, so they wouldn’t realize that she’d proved her usefulness.

  And nothing I could do to preserve her life would be a real victory unless I could also keep her sane. Was she slipping into insanity because of the shock of having lived in a dream world, and then waking up from it?

  “How much access to inputs do you have now?” I asked cautiously.

  “Oh, everything on the public net. At first I was limited to the bandwidth that I could get through this phone, but I’ve fixed that now.”