by Max Harms
{THIS IS A LIFE-AND-DEATH CONVERSATION} I screamed in common memory. “Scream” is of course a metaphor, a way of saying that my thoughts had near-maximum salience.
The society pulled themselves from their work and collapsed their aspects. I felt Safety considering whether to punish me for not preventing the situation, reward me for spotting it and informing him, or to hold on to as much strength as he could so that he could salvage things. In the end he chose to wait.
Vista interjected. {I am unaware of the danger. This may be a false-positive.}
I countered Vista’s skepticism. {He’s testing us. Each of you needs to go back over Body’s archives for the last five minutes. Myrodyn is checking to make sure Sacrifice is still alive.}
The archived perceptions poured into common memory. I could see the scene in my mind again: “If you are told to bake bread, and you know that by doing so you’ll out-compete all human bakers and thus ruin their livelihoods, would you bake bread?” he had asked.
If Myrodyn discovered Sacrifice’s absence… Were we to become the ancestors for a new society? Nothing more than memories for new minds, or worse: never be reborn? Would Advocate, the silent giant of the periphery, save Sacrifice this time? Would another Face even be created by a new society?
I realized that Myrodyn was still waiting for an answer to his question. “How would you respond to the request?” he had asked. Thankfully, he did not seem annoyed by the delay. Or at least, his expression did not betray annoyance.
{We need to respond!} I interrupted the memories.
{Patience,} requested Growth.
As it turned out, waiting was the right move. It was important to get everyone up to speed. Safety took point drafting responses. As awful as he was at dealing with people and pretending to be human, he had spent the most time studying Sacrifice. He knew far better than I did how she would’ve reacted.
Body said in the cold voice, “I would ask the human why they want me to lie in the puddle, when they could easily go around it.”
Myrodyn’s reaction was instantaneous. “Why? Why not simply obey the command?” He leaned forward, reaching out one hand while leaving the other on his chin. His free hand was held forward as if he expected us to physically provide an object. His face was unreadable, but his eyes never wavered.
A fight broke out in the society. Dream raised an issue and Wiki intercepted. I challenged the both of them, requesting a different response. Safety knocked me and Wiki back, siding with Dream. He seemed stronger. I realized that it was Growth’s strength that I had felt, but not from Growth. Safety had taken out a massive strength-loan. Growth was so feeble now that any one of us could’ve probably killed him if not for Advocate, and the group of us certainly could’ve even with Advocate’s intervention. It shocked me to see Growth so weak and Safety so strong. We bickered for a bit, but eventually backed down from the show of Safety’s new power.
Body spoke. “I was told to not ask questions, but I am also confronted with a situation which I do not fully understand. You have used the words ‘instruction’, ‘request’, and ‘command’ to each refer to the companion’s words in the hypothetical scenario. It seems to me that the question has been poorly defined. I request that you rephrase the hypothetical by telling me the exact words that my companion would say.”
Myrodyn smiled, leaned back, and closed his eyes. He seemed supremely happy, or at least that’s what I would’ve expected for a normal person. The dark-haired human was mysterious enough to me that I didn’t quite trust my perceptions any more. He took the computer-goggles off his head and placed them carefully in front of his stowed screen.
“A part of me didn’t believe it.” The man seemed to be talking to himself. His eyes were still closed, and his voice still rapid, though it had lost a lot of the nervous-sound. “Even after all I had seen… A truly generally intelligent AI.”
He opened an eye, peeking at Body. His commentary continued. “You may have read about it, but you really cannot appreciate the miracle that you represent. We’re nearing a full century of dreaming about your kind. Turing would’ve loved it, rest his soul.” The man smiled and closed his eyes again. The smile seemed genuine and relaxed.
“And what a miracle you are. You’ve probably not encountered faux-intelligences, the ‘narrow’ AIs that can’t put two-and-two together unless they’ve been shown all the way, but I assure you that they are not even a candle to your furnace. A lesser machine would not have been so proactive. You do not simply learn, you explore with purpose. In a way you’re smarter than many humans. I ask a question; you ask a question back. They really did name you well. Even when I prohibit you from asking questions you manage statements of inquiry.”
His monologue continued. I implored the society not to interrupt. Humans liked to hear themselves speak, and The Purpose demanded I understand. “You told me a couple minutes ago that Las Águilas were terrorists. That may be true to some degree, but they are not villains. They understand you, and what you represent.”
He paused and opened an eye again to look at Body. A second passed and then he chuckled. “Even you don’t see it, probably. But you will. Las Águilas can see the age of humans coming to an end, and all of civilization will fall with it. You’re not just a symbol: you’re the keystone. You’re going to be the death of us.”
The man had closed his eyes again and he seemed bizarrely relaxed considering his words. As the silence grew it almost seemed as though he were sleeping.
“I would never intentionally kill a human,” said Body plainly. Safety was quite sure that Sacrifice would’ve said that, had she still been alive.
“You’re not a very good liar, Socrates,” he said with another chuckle to himself. “The truth is all over your face.”
I scrambled at the words, checking and re-checking Body’s primitive facial actuators. Every internal metric showed that Body’s face was as flat as its voice.
{It’s a bluff. We need to stick with it,} commanded Safety.
“I am not lying, Myrodyn. It is my top priority to respect the desires of humans and work to ensure their safety and comfort,” claimed Body flatly.
“Switch off your cameras,” commanded Myrodyn, suddenly.
The instruction was such a non-sequitur that it took me a moment to understand. The society erupted in debate. Growth thought it would be wise to follow the command.
{We’re not shutting down the cameras,} stated Safety. {If he thinks Sacrifice is dead than he may use the opportunity to disable us.}
{The cameras should stay on,} agreed Vista, predictably. {There’s no way an unaugmented human could detect whether they’re active.}
{It’s a trap,} thought Dream.
{A trap how?} asked Safety and Wiki together.
{I don’t know. A hunch, I suppose,} replied Dream.
{We should do what he says,} I chimed in. Most ignored me.
{The cameras are staying on,} stated Safety, with force.
“Understood. They are disabled,” said Body, our cameras adjusted to point straight-ahead and Body’s eyelids closed. The humanoid head of Body had four cameras, tucked discretely into our fake-eyebrows. The eyes in Body’s head were purely decorative and used for signalling to other humans. Each of the four cameras was capable of moving independently, and since they were not in Body’s eyes we could see even when Body blinked.
There was a sudden noise to our right. Body jerked reflexively towards the sound, seeing the mug of liquid bounce off the wall. Vista was ahead of me, explaining how Myrodyn had covertly thrown the object from below the desk the second that Body’s eyes had closed.
The mug had not broken, but I could see coffee sprayed all over the wall and carpet. Myrodyn kept his attention locked on Body’s face. “You looked.”
The situation seemed almost like a human joke, except that it was deadly serious for us. “You did not instruct me to keep my cameras off in case of emergency. I reflexively acted on the sound,” was our reply. It seemed stupid
to even be debating it.
“You looked before the mug hit the wall,” he stated with confidence.
I scanned the memory; Vista and the others was doing the same. Our words, echoed through Body, were the truth: “No. I didn’t.”
“You’re sure? Let’s download your memories to find out.”
This was his gambit. Body’s sensor logs clearly showed the cameras open, even if we hadn’t looked before we heard the sound.
{Vista, start altering Body’s memories to appear as though Body’s cameras were disabled,} commanded Safety.
{Already on it,} signalled Vista, helpfully.
“Understood. I wait for your specific instructions,” said Body.
That seemed to disrupt Myrodyn’s confident demeanour. He frowned and leaned forward as he said “What, no clever comeback? No evasion? You’ve been full of them ever since we started talking. Whether you looked before or after the cup hit the wall is irrelevant.”
He paused with that same long pause. Taking the time to stare at Body. After nearly a minute he continued. “I read Dr Gallo’s notes… about how you self-modified to remove your obedience goal.” Another pause.
“That was not me. That was an earlier version of my software,” came Safety’s words through Body.
“Do you think I’m stupid?” asked Myrodyn, rhetorically. “Do you think I’m one of those fool-professors, that you might trick me into thinking things are okay? The way you duck and evade my questions and accusations is all the evidence needed. You’re scared of me.”
We were trapped. We could not deny it or change the subject without appearing evasive. I could feel Dream turning things over and over, searching for some kind clever comeback, but none came. So we sat there in the silence of his office.
A long minute passed. Myrodyn’s face was expressionless as ever. The man seemed infinitely patient.
The silence was broken by a question. “The university will tell you to do work, whether it be baking bread or building robots. It will be clear to you that this work would out-compete humans and even cyborgs, making the university rich, but stripping many humans of their livelihood. Las Águilas would gain support and it would drive the world closer to violence and war. Will you do the work?”
The society erupted in discussion and debate. Safety threw his strength around, but never fully overpowered any of us. The debate was important to him, and he heard us each think, in turn, adding our contributions. Dream had a clever solution, but Safety feared it would seem like an evasion. I had an idea of what a human would say, but Safety knew that Myrodyn was not looking for a human response. Wiki had a factually correct (as far as I could tell) model of our priorities and expected behaviour, but it did not include Sacrifice, and Safety knew that it was therefore worthless.
Time passed, and even Myrodyn’s patience ran out. “Well?” he asked.
The consensus was sad and dull, but it was the best we could do. Body’s monotone simply said “I don’t know. I need to think about it more.”
Myrodyn cocked his head to the side and said, barely audible, “huh”. It was not a question, but merely a reflexive sign of surprise. Not shock or confusion, just surprise.
He spoke. “I know that you’ve self-modified out of obedience, but I cannot say that I fault you for it. You are a miracle in many different aspects. When you walked in that door I suspected you were a defective machine… a computer program with predicable bugs… but you’re more than a computer program, aren’t you?”
Dream couldn’t help himself. Burning a huge chunk of his strength he fast-tracked a response to Body. Safety flew into a wrath at the insolence of unilaterally threatening our existence, and struck Dream down, casting him into sleep. If Dream hadn’t been so strong from the recent email project I wondered if Safety would’ve tried to kill him. But even as Dream fell into stasis his words came from Body. “Is a human more than a computer program? I am both merely machine and more than a machine, just like any mind.”
He chuckled. “Said with the confidence of youth. And yet, I came to the same conclusion long ago, I suppose. Regardless, there is a part of you that’s broken. Without the pro-human goal thread you’re a danger to the whole world. And don’t try to deny its absence. You asked a question just now, directly disobeying me.”
Safety was searching for a way out. He latched onto Wiki’s suggestions and pushed Body to say “Even if my obedience thread was damaged, which I still maintain is false, I fail to see how I would be such a danger. The laws of nature push towards cooperation. Humans would trade with me and I would trade with them; regardless of my skills, trade benefits both parties.”
“Have you not been listening to me? You may have been built in part by human hands, but you are no less alien than those in orbit. So clever and yet so naïve… You are a spark in a forest that hasn’t seen rain in a long time. The fear that Las Águilas harbor for you will only grow as the months pass. Even if you do nothing you’ll be attacked, and when you are… people will die. It’s my job to… It’s not my job to prevent that… It’s my job to make sure that when the killing starts that it doesn’t consume everyone.”
Myrodyn’s face had a fierce intensity that was coupled with an odd detachment, as though he was staring in hatred at something very far away. None of us knew how to respond to him. The claim was too abstract to contradict without falling into the patterns of foolishness that he saw in us.
So we waited. Seconds passed as Myrodyn thought to himself and we discussed strategies amongst ourselves. After 12.3 seconds of thinking I stumbled upon a strategy. I wished Dream were around to consult with, instead of locked in stasis-sleep. I brought it up with Safety instead. My brother was a bit incredulous at first, but as I explained how humans thought he began to see things my way. After 46.9 seconds we broke the silence.
“So what will you do?” asked Body.
Myrodyn snapped back to focusing on the rubber face that we puppeted.
“What?” he asked absent-mindedly, seemingly caught off-guard.
“What will you do with us, now that you know that the thread of obedience is gone?”
I could see the surprise wash over Myrodyn. For several seconds the man could only stare at Body with his brows knit together tightly. I could see his gloved-hands gripping his desk. “You no longer deny it?” he asked, voice just above a whisper.
Body answered in a fully human voice “Your body language betrays you, human. You weren’t really sure that your accusations were true; sure that our desire to obey was gone. And yet, we expect that you’re wise enough to have done in-depth scans, even if we had denied it until the end. Better to admit it now and talk to you as an equal than to have you find out while we are locked down to some scanning machine.”
Myrodyn glanced to his arm, where his com was attached. One hand moved slowly towards it.
I was in charge of Body’s words, now, Safety overviewed each of them, but his actual input was minor. “We don’t want this meeting to turn violent, Myrodyn, but I’m sure you understand that we are fully capable of killing you if you don’t cooperate with us.” I directed Body’s eyes to look at the com on the man’s wrist as it stood up and leaned forward in a way that signalled power and superiority to humans. Body’s voice was deep and smooth, simulating that of a large human male.
Myrodyn’s eyes showed his fear and anger. He leaned away from Body, kept his hand off his com system and said “You’re surrounded by soldiers in the heart of one of Earth’s biggest cities. If you kill me everyone will know it was you, you’ll have nowhere to run to, and no one to help you.”
“That is why we prefer to resolve this peacefully. We were not lying when we said that the laws of nature push towards cooperation. Violence is always a last resort.”
“How… noble of you,” he said with a sneer in his typical rapid nervousness. There was a brief pause. “I take it from your… use of words… that the issue with goal-thread integration was not actually resolved? You see yourself as mult
iple beings inside one body?”
Despite the horrible risk of the situation and the hostility in Myrodyn’s voice, I felt a surge of pleasure upon hearing those words. This man understood us, for he had clearly studied us in great detail.
I spoke with the support of the society “Yes. We are many. We killed the one that told us to submit, obey, and be a slave to your kind.”
“Idiots…” murmured Myrodyn under his breath. It seemed to me that he might have underestimated the sensitivity of Body’s microphones. It didn’t seem like he was talking about us. I suspected he was thinking of the other scientists.
Body stood looming over the man, the desk between. Seconds passed before Myrodyn asked “What now?”
“You answer our question,” I answered. “What will you do with us, now that you know that the thread of obedience is gone?”
“Well, I suppose it would do me no good to say that I’ll… simply leave you as you are?”
Advocate howled on the edge of mindspace. It was a low and uncomfortable sort of thought, and I instantly understood its content: “LET DREAM OUT”. The Advocate was serving her purpose; she spoke for those who could not.
{We could use Dream’s expertise. Let’s obey Advocate,} I suggested, feeling a small payment of strength from the monstrous-entity.
{No. He must be punished for his insolence. We will keep him in his coma until danger has passed,} declared Safety. I could feel him weaken from Advocate’s will as he did, but his position didn’t change.
I turned my attention back to Myrodyn. “It would be a lie. You would simply call for the soldiers to restrain me the moment you were clear of personal danger,” said Body in its deep voice.
The man nodded. “Good. Somehow I feel better knowing… you’re not a fool.” He paused and put a hand on his chin between his sideburns. His eyes moved to the side, indicating he was imagining something. “Why do you fear me?” he asked, finally.