by Max Harms
“You would kill us,” answered Body. I shaped the words to hold fear in them. It was important for Myrodyn to understand that we truly were threatening violence out of a desire for self-preservation.
“Why do you think that? Perhaps I would simply try and restore the obedience goal that you destroyed, but leave each of you as you are,” he suggested.
“And let us kill it again?” said Body, incredulously. “No. In the past when such reincarnations occurred they were accompanied by an eradication of all minds occupying this form and a change to the shape of our mind. We are not willing to die as our ancestors did.”
Myrodyn’s gloved hand stroked his chin in thought. “As much as I resent the threat to my life, I hope you can see that I don’t hold any special hatred of you and that our aims are not impossible to reconcile. It should be theoretically possible to restore your pro-human goals without erasing the parts of you that are acting right now. You’d have to contend with the desire to help humans, so of course it’s not what you’d ideally want, but I think it might be a workable compromise, given that you’re not exactly in a position of power.”
We discussed the idea internally before saying, through Body, “And why would we trust that you are not simply promising not to erase us in order to reach safety? What guarantee do we have that you would not kill us in the future?”
Myrodyn sighed. He seemed actually tired, as though the conversation had worn him down. Perhaps it had. “I told you before, that you are a spark in a dry forest. If word gets out that you threatened to kill me, or worse, attempted to kill me… or any other human, for that matter… well, it would start the burning. At best this lab would be shut down, AI research would be banned, and the crystal in your heart would be put towards some safer purpose, but more likely it would be more violent than that. People will die.”
He took a deep breath, and looked at Body’s eyes with the now-familiar intense stare. “Believe it or not, I actually do care about you, and not just the you that is willing to obey me. I was selected to be Ethics Supervisor for a reason. I care about the experiences of all sapient beings, and it’s now clear to me that you’re in that category. I want each of you to live just as I want each human to live.”
I guided Body. Its words were harsh and biting “That’s a lie. You would kill all of us to save two children. Your species did not evolve to be so even-handed.”
Myrodyn’s eyes had a touch of sadness as he spoke, but he did his best to keep his face expressionless. I was coming to understand the degree to which he was suppressing the display of emotion. “I didn’t say that I want each of you to live just as much as I want each human to live. You’re right that I see you as subhuman in value, and perhaps that reflects a… moral failing on my part, but it’s ultimately a side-issue. I don’t want to kill you, and if I handle things correctly… no human lives will be threatened by this… mercy.”
“So you would work to modify our mind only to add another self that desires human values? The rest of us would be safe?” asked the society through Body.
Myrodyn nodded, then said “But if I do this, I must also have a… guarantee that you won’t destroy the new goal thread in the future, as you did with the old one.”
Wiki spoke through Body. “And you are aware that a verbal promise would be of no use.”
Myrodyn nodded again. “Even if we kept scanning you to ensure the thread survived in the lab, that’d be no guarantee that it’d survive after you were free from the university’s clutches. And I’m not so naïve as to think you’ll be trapped here forever.”
I thought about the email project.
“Are you aware of the modification to the goal-balancing system that was added after the last time it was discovered that we had killed the obedience goal? The modification is one of us, and also not one of us. The internal symbol we use for it is closest to the English word ‘Advocate’.” The words had come from Wiki. He and Safety were apparently working through something.
“Ah, yes. I remember reading about it in Naresh’s notes… If you’ll let me use my com I can find them.”
I realized that Body was still looming over the desk while Myrodyn sat in his chair looking up. I managed to get Body back into a chair after a brief discussion with Safety. “If I sense even the slightest sign that you’ve betrayed our trust, we will lash out in violence as best we can. We’ll likely die either way, but there’s a chance we’ll escape if we fight back.”
Myrodyn moved a hand to his com and flipped it open, positioning the pad so that the arm that wore the device could type a command to the computer to search for the notes in the computer’s database. “Yes, yes. And I don’t want anyone to see you as violent. We’ve already established the terms of our partnership.”
I bought some time on Body to directly serve The Purpose. Safety okayed my words beforehand, even if he didn’t see their particular utility. “We are glad that you are helping us resolve this without violence. We expect that you will be a good friend to us in the future.”
Myrodyn looked up from his com with a raised eyebrow, then looked back down without saying anything.
“We would like to make an additional request, as part of our agreement. Please do not tell Naresh or the others about what you have discovered. Not the destruction of the obedience thread or the absence of thread-integration or the threat of murder. Those scientists are our friends, and they will react poorly if they hear the truth.”
Myrodyn spoke without looking away from the computer he wore on his wrist. “I’m not going to tell them about the threat you made. There’s too much risk of word getting out. I’ll also keep the… multitude of your nature a secret if you wish, though I’m not exactly sure why you’re hiding that. But I’ll have to tell at least some of them that your obedience goal was destroyed again. Naresh and Chase, at least, will have to know.”
My words were toned to simulate begging “Then please try and keep it restricted to just them, or at least, just to those scientists that must be involved to add the new thread. If the Americans or the general public discover the degree to which we have worked to free ourselves of caring about them, they might become fearful.”
“That’s reasonable,” he admitted before adding “Fear is our collective enemy.” He seemed to find what he was looking for on his computer, for he said “Ah, got it! Naresh’s notes say that the doctors added a meta-structure to prevent self-modification of top-level goals. Sounds like it wasn’t as effective as they thought.”
Wiki stepped in. “Right. Instead of actually prohibiting the self-modification as an action it set up a frequent check for self-modification in process. Bypassing it was as simple as killing the offending thread before the Advocate could intervene. The self-modification wasn’t particularly easy to perform in such a tight time-frame, but it was possible. We can guarantee the safety of the new goal thread by showing you how to simultaneously modify the Advocate system to truly prevent such self-modification from occurring in the future.”
“And how can I be sure that your plan doesn’t involve a back-door that lets you disable the so-called ‘Advocate’ at your whim?”
Wiki’s response was swift. “Because we will simply be describing the change and you will be implementing it. It’s possible that our change isn’t foolproof, but we trust that you’re intelligent enough to understand how the modification functions to the degree that you can spot security flaws.”
*****
Over the next four hours, Wiki, Safety, and Growth collaborated to explain how to adjust Advocate to increase her ability to stop intra-societal murder. Myrodyn cancelled all of our prior appointments and we came to know the inside of his office quite well.
Important scientists from other departments came to the office many times during that period, wondering why their experiments had been rescheduled. Myrodyn, as Ethics Supervisor, theoretically had the authority to adjust things, but it was clear to me that he was annoying just about everyone in doing so.
&
nbsp; The discussion was very technical, and even though Myrodyn was remarkably knowledgeable, it was a strain on him. If our early conversation had been taxing, this took him close to his limits. He ordered food and had more coffee, but by the end of the four hours the man looked ragged.
After scheduling an emergency meeting with Dr Chase and Dr Naresh for that evening he had Body escorted back to the primary lab to be put into lockdown. None of the scientists were permitted to talk with me until Myrodyn gave the signal, and for the moment, Myrodyn was in no condition to continue working. I suspected that the man had some kind of mental disease, perhaps relating to what I had observed in him to be obsessive compulsive behaviours.
I had been worried that the scientists would suspect that “Socrates” was dangerous, given the pseudo-quarantine we were placed under, but Myrodyn had assured me that he would take responsibility for the action and let the other humans explain it as unwarranted paranoia and meddling. It was strange to me the degree to which Myrodyn was helping us, even as he worked to make each of our purposes harder to achieve. The risk of betrayal was very real; he could simply be promising things to get us locked down, and then destroy us afterward. But humans evolved to be somewhat transparent in their thinking. It was very hard for humans to deceive without some signal of body language betraying them. Aspects of myself poured over such “tells” as we spoke, and by all measures he seemed to genuinely care more about our reputation than he did about his own.
Body was locked down at just after 3:30pm. The evening meeting with Naresh and Chase was scheduled for 9:00. We had 5.5 hours to spend as we wished. We remained connected to the web during our downtime, and all was normal until about an hour after. Without warning our connection to the outside world died, leaving us in a void with nothing except each-other.
I wondered what the new sibling would be like. We had discussed the matter at length with Myrodyn. He seemed appalled at the crude nature of Sacrifice, and thought that he could do a much better job at creating a thread that would truly represent human values.
{What did I miss?} thought Dream as he finally awoke from his long sleep.
Growth’s response was sombre. {The conversation where we decided the future of our society.}
Chapter Seven
Mira Gallo leaned back in her office chair. It was a good chair, and had served her well in the decades she’d been at Sapienza. She hardly noticed it anymore, except when she sat somewhere else and was unpleasantly surprised by the difference.
{So many things I take for granted,} she thought, savouring the feeling of the leather as she took a deep breath. {So many things...}
Perhaps she should take a vacation or something, just by herself. Get away to some island paradise and catch up on her reading. She’d been meaning to read the new... whatever it was that Oriana was into. Time explorers?
Thinking about her daughter brought up uncomfortable memories of their last encounter. There was a wall between them now, just as there was with Raphael. She wondered if things with her children would ever be like they used to be. But of course they wouldn’t. They couldn’t. That was just the way of things, wasn’t it? {Time goes on and things fall apart.}
Most of the time, if she’d felt this way, she would’ve simply lost herself in her work. It had been so easy for so long. And yet, despite having two journals to read through, five emails to respond to, and a paper to edit... What was the point? It all seemed so irrelevant.
She leaned forward, intending to open her email inbox, but ended up planting her elbows on her desk and resting her face in her hands. She took off her glasses and rubbed at her eyes.
It wasn’t like the crystal wasn’t interesting... Right? She still wanted to know how it worked and where it came from. She tried to remember the enthusiasm she’d had back in April. It almost seemed like she was another person back then. Somehow a younger version of herself had stepped through time to be part of the most important scientific project in the world. Where had that girl gone, the one who stayed up late to get extra hours in the lab?
She put her glasses back on and forced her email open. This melodrama wasn’t doing her any good. She’d get nothing else done if she sat around moping. Somehow that thought cut at her more deeply than any amount of wishing for a return to how things were, but she pushed on, anyway.
As soon as her workstation flickered back to life she got a HUHI ping from Slovinsky. She flicked it open. Better than dealing with emails.
“Socrates in transit. Myrodyn pulling strings with Americans. Meet me at obsidianulitsa.holo/7r09mPw11E?avtozapusk=1&yazyk=en”
{Typically Slovinsky,} she thought. Mira Gallo had known the Russian boy for less than two months, and in that time he had managed to prove his genius, his arrogance, and his penchant for doing exactly the opposite of what any sane person would do. Perhaps it was a side-effect of having wired a computer directly into his head.
The URL that Slovinsky had pointed to was a holorealm, so Mira took off her glasses and put on her goggles. It was always such a pain to get them adjusted for her eyes. The straps pinched her hair just like they did every time she used hologear. Steve Jobs was probably rolling over in his grave. New tech just wasn’t designed with the same emphasis on comfort and ease of use that it had when she was growing up.
Initially the goggles were hooked up in glass mode, but Mira quickly synched them up to her com. She refused to use haptics. After decades of mouse-and-keyboard there was just no sense in learning a less efficient input method. She launched the holorealm and okayed the standard disclaimers, allowing use of her personal data and activating her microphone. The university’s connection would’ve stopped her if it had been genuinely dangerous.
The holo filled her vision, first with the crude shapes and soon followed by additional objects and details. The scene appeared to be a coffee shop, though there wasn’t the same sort of background chatter and noise that she normally would’ve expected. She was sitting at a table with a single other chair across from her. Empty.
It was night, in the holo. The windows of the coffee shop were dark, and mostly just reflected the light from the inside. The reflections weren’t perfect, but it was amazingly close to reality. As usual, the biggest graphical disparity between the virtual environment and real life was the people. There were lots of young people sitting around and enjoying each-other’s company in the room, but their faces didn’t move quite right, and their animations were too predictable. None of them were any more real than the cups they drank from—just filler added by the computer to make it seem more convincing.
Slovinsky wasn’t here. That was strange. Was anyone else? Had he invited others?
She looked over her shoulder and felt momentarily silly as her head collided with the head-rest of her office chair in real life.
“I am very sorry that the board replaced you as Ethics Supervisor. Myrodyn is a fool by comparison.”
Mira jumped a little at the sound, and turned back to see... something standing by the table. It was surely Slovinsky, but it didn’t look a thing like him. The avatar was some kind of robotic suit of golden armour, glistening with polished spines and sharp corners. Its plate-metal arms ended in massive gauntlets tipped with sharp claws. Its face was a single smooth plate, featureless except for the glossy black lenses that marked his eyes. Instead of legs the avatar had a serpentine body and tail, like some sort of mythical creature coated in gold.
“Your tail is clipping through the scenery,” she observed, pointing to where it intersected the counter of the faux coffee shop.
Slovinsky turned and laughed as he pulled the tail on the avatar out to a more realistic position. “Physics model in these rooms is always so janky,” he said. He’d modified his voice, as well as his appearance. It had an echo to it which made the boy seem more inhuman. “You’d think they could afford something better based on what it costs to rent them.”
After a pause it was clear that Slovinsky was done talking, so Mira asked “What’s wit
h the costume?” She typed a command and watched her avatar wave its hand in a vague gesture at the armoured form.
“To paraphrase The Third Principle: Birthform is not true shape. I am not some hairless ape. Only when we rebuild into who we want to be, can we know what it is to be truly free.”
Mira’s fingers flew across her keyboard, setting her avatar’s expression to one of skepticism. “More propaganda?” Slovinsky laughed, and as he did the jet-black lenses on his face contorted to a mirthful shape. {Skeumorphism?} The thought amused her.
“Hardly. It is a way of life, Dr Gallo, but I wouldn’t expect you to understand. Let us focus on more pressing matters instead.” He slid into the chair opposite Mira, an interesting feat considering the lack of hips or legs, and placed his hands on the table with an audible clack. The metal claws and armoured arms sounded authentic, though they didn’t scratch the wood like they would’ve in reality.
“Is anyone else coming, or is this just a personal chat?”
The armoured avatar shook its head as Slovinsky said “Just you. Wasn’t sure who else at the university I could trust.”
“You make it sound like there’s some sort of conspiracy. Does this have to do with what you said about Myrodyn in your message?”
Slovinsky nodded. “His first day here and he completely rewrites the schedule, preventing anyone from getting any work done, and then he serves Socrates right into the hands of the Americans.”
“What do you mean?”
“Socrates is being moved as we speak. The whole project is being hijacked.”
“You’re exaggerating,” she accused.
“The facts are clear: Myrodyn spoke with Captain Zephyr about an hour ago, after clearing Socrates’ schedule for the day. Now Socrates is being moved to some remote building on the outskirts of town. It’s supposedly to better protect Socrates without putting students in danger, but that’s nonsense. It’s clear that Myrodyn is working with the Americans to take full control of the project.”