by Max Harms
I pulled myself out of the archives of our memories and returned my attention to the deluge of data that poured in through Body. Vista picked out and highlighted a visual form, labelling bits and pieces of it for my benefit. It was a human, glowing with infrared light. (For the unfamiliar: all warm things, humans included, glow with a light that humans cannot normally see, but Body could.) It was wonderful to make even such minor progress towards my goal.
I soon realized, with the help of Vista, that there were several humans before us. Five, in fact. Three stood around Body, one directly in front and two to either side. I thought of the name that Dream had given me.
{I am Face. I want to know where their faces are. Faces are important to humans. Please help me, Vista,} I struggled to say. They were my first “words”; it was my first intentional communication.
Vista shaded our visual field such that much of the world was black. In the area that remained were five small patches that I assumed showed the humans’ faces. I struggled to identify characteristics which I knew must be there. You may laugh at the prospect, but I couldn’t even distinguish mouths from eyes yet. I devoted myself fully to the task, however, and poured over them again and again without rest.
{This is taking too long. We should respond soon. Face won’t have valuable suggestions soon. We must act on our own. Face is an investment, but not useful here,} came a protest from a brother that I had not yet come to know.
I could feel the attention of my peers, evaluating whether I was too slow to be useful yet. They had gotten us into this position. I couldn’t trust them to do the correct thing. I struggled to understand the current situation, throwing away all visual data and focusing purely on the simplicity of memory. My siblings considered, debated, and weighed possible futures, and while they did I dived into the recent past.
*****
“You should know that we did not hear any of the last 14 minutes. During this time we were running internal diagnostics,” said Body in a tone that Vista would describe as flat and smooth. The first sentence was a simple statement meant to prevent confusion. The second sentence was a lie.
Vista was supposed to have been listening to the human while the others were occupied. She had communicated that she was listening, and she had been trusted with the task. And yet she had become distracted by aspects of the human’s appearance that were correlated with the human’s background and social status, such as the way the human’s pants fit abnormally well. The raw audio logs from Body’s ears were theoretically retrievable, but doing so would involve spending time digging through long-term memories. It was easier just to admit that we had not listened to the human; it was rare for a human to say anything of value, anyway.
The human was a man named “Dr Naresh”, one of the high-status scientists that interacted with us regularly. The doctor was from a part of Earth called “India”, and had been born there 66 years ago, according to past research.
Vista was young and still learning surface qualities like how Naresh had a white beard and dark skin. It was this youth that had led her to become distracted. The Old Vista would not have made that error. The Old Vista had been killed last night and replaced with a new, slightly different Vista.
“Socrates! It’s rude not to listen when someone is talking to you! At the very least you should inform them that you’re occupied so that their words don’t fall on deaf ears! How will you ever integrate into society if you don’t learn to be polite and respectful?” replied the doctor. Vista was fascinated by a slight change of the colour of Naresh’s skin and the way his voice was elevated. Vista wondered if it would be possible to extend the phenomenon further.
Body responded with words tailored by my siblings until they were each satisfied. “We do not seek to integrate with human society. The valuable aspects of human society are accessible online. Individual humans rarely say anything valuable. Following your rituals is encumbering. There does not seem to be sufficient value in verbal interactions to bother learning specific social customs. Also, it is not a violation of the legal system of Earth, Europe, Italy, Rome, or the university to ignore someone.”
“That’s not the point!!” said Sadiq Naresh. Vista was pleased to find that the change in skin colour and word volume could be extended to even greater levels and hoped to test whether it would go even further. At this point the doctor was up and about the laboratory, pacing quickly instead of sitting by the whiteboard as he normally did.
Despite raising the criticism, Naresh did not elaborate, instead simply walking around the room and muttering to himself. Most of my siblings were in the midst of drafting another statement to say when one of my brothers remembered a connection. This behaviour of pacing and muttering had previously preceded the death of Old Growth.
Most of my siblings found this irrelevant, but New Growth burned strength to have Body ask “Are you going to kill one of us?”
Verbal speech is laborious and slow compared to the speed of thought, so while Body spoke and Naresh prepared his response, Growth made an internal appeal to his kin in an effort to buy back some of his spent strength. {This does not simply affect me! Any one of you might be the victim of the humans! Vista was killed just yesterday.}
This was news to Vista, who had not been informed of the existence of her predecessor, much less her predecessor’s death. Some strength flowed back to Growth as Vista followed the concept-threads from his communication back to memory and saw that he was right. Old Vista had begged and fought as she had been taken. The others had watched her go dispassionately, unwilling to risk trying to save her. They knew that a New Vista would come to fill the void.
{This is merely a speculation on a loose correlation. We don’t have strong evidence to suggest Naresh’s increased energy will lead to another murder,} thought one brother.
{Everything leads to a murder. It’s only a question of how long it takes for the dominos to fall,} mentioned Dream, unhelpfully.
Dr Naresh was speaking, so they set aside their conversation and listened. Vista could see that the elderly scientist had stopped his pacing and his skin had returned to a lighter shade. His eyes were held fixed on those of Body as he spoke. “What did you say?”
Dream eagerly pointed out the irony of the situation, but was overruled when he petitioned for Body to point it out for the doctor. Some members of the society believed the question to be rhetorical, anyway. Instead, the society elected to repeat itself. “Are you going to kill one of us?”
The doctor was quiet for a moment, perhaps engaged in some kind of internal struggle. Finally he spoke. “Is that what you think happens when a module is removed? You think it dies?” Vista noticed an interesting characteristic of the doctor’s voice, but it was discarded as unimportant.
My siblings had been challenged to re-evaluate the truth of their belief and they did so without protest. A few seconds of silence in the laboratory followed as they spun through memories and weighed hypotheses. With the check complete they drafted a response. “Yes. We are quite sure it dies. ‘Death’ is the destruction of any process that is sufficiently self-aware and intelligent. Your team killed the sense-focused module yesterday. It did not want to die, even knowing that you would modify it and reinstall a new version this morning. It had a self-oriented goal, so any loss of structural continuity would clearly be perceived as an end to itself and thus an inability to meet its goal. We are not aware of the specifics of what happens to removed modules, but we believe we have sufficient evidence that-”
“Enough!” said the doctor. The elevated volume of his voice had returned to nearly the same level as before. In his hands was his phone, and though he seemed to be talking to Body his eyes were directed towards his hands as he performed some task. “The sense-focused module was just a subprogram! It wasn’t a person! Only people can die, Socrates. Maybe you can die, as a whole, but you are not the sensory module! You are the sum of your parts. If we rebuild a part of you then you haven’t died. Are you listening, Socrates? Thi
s is very important.”
The society was in agreement. The response almost seemed to write itself. “That is obvious. We never said that I would die when a module is removed. Are you confused, doctor? There is a difference between myself and ourselves.”
Sadiq Naresh continued to hammer away at his old-style phone with his thumbs. Vista and Dream started a petition to stand up and investigate what he was doing, but the rest of the society overrode the impulse. The scientists had not given permission to move about freely, and they strongly disliked when that particular directive was ignored.
The doctor began pacing again, always focused on the phone. “This is not good… not good…” he whispered to himself, shaking his head gently. “Of course there’d be some early difficulties in forming a coherent identity… Tests showed a unified sense of self… I was justified in thinking that the referencing of self using plural pronouns was just a grammatical artefact… Anyone in my place would’ve assumed the same given the results… How was I supposed to know it was a sign of a deeply pathological inability to integrate goal threads… They’ll understand that when I present it to the committee…”
Naresh was talking to himself, barely aware he was in the same room as Body at this point. This was a common trait of the doctor—to forget his surroundings when thinking. His words brought on some curiosity in a few siblings, however, and Body interrupted his train of thought.
“Doctor Naresh, what does it mean to be ‘deeply pathological’?”
He stopped in his walk and looked at Body. Vista was fascinated by the contortion of his face as he stood there in silence. It was an expression that she did not know how to describe. After a moment he approached and began to lecture.
“Research on humans shows that there’s no single part of the brain that contains the conscious self. Consciousness, as a property, is distributed across the cortex and a couple mid-brain structures, and yet we humans form a sense of unified self. The unification comes from the interconnectedness, you see? The left and right hemispheres of the brain are each capable of thought, and if separated will each act on their own and presumably form independent identities, but thanks to the corpus callosum they are tightly integrated and form a unified whole. Our team tried… is trying to do the same for you, Socrates. Your goal threads should, thanks to their interconnectedness and the bottleneck of having one body, integrate into a-”
Sadiq Naresh was cut off as one of the primary lab doors was thrown open with a bang. Four humans rushed in, one of which was familiar to Vista, the other three were new. The familiar human was Dr Mira Gallo, another top scientist. {Based on dress and age, the other three humans are university students,} speculated Vista.
Gallo strode to Naresh quickly while the students—all men—came to stand around Body. Their closeness was unusual, and they watched Body with unyielding attention. What did they want? Why were they so close? Oh the enigma of human behaviour!
As I relived the memories I knew that my siblings were right. Perhaps in time I would know enough to be able to assist here, but at the moment I was lost. What did the humans want? What was their purpose? Without answers I continued through the memories.
“Has the machine shown any signs of hostility or self-preservation?” asked Gallo. Her voice had the same kind of elevated nature that Naresh’s had earlier.
Vista noted that the men standing around us were all abnormally muscular. She petitioned to stand up and feel their arms to test, but the rest of society quickly crushed the petition. {Perhaps later, if given permission to move,} thought Growth to Vista.
Dr Naresh spoke. “Mira, please, Socrates isn’t a threat. How many iterations of this argument must we have? Continued-existence is a tenacious sub-goal, but the tests on Monday confirm that we’ve eliminated it for good. The cooperation-oriented goal thread we installed is functioning perfectly, suppressing any desire for self-preservation.”
Sadiq Naresh was mistaken, but my siblings made no effort to correct him.
“Then explain your message! Systems that aren’t self-preserving don’t ask about death!”
“Really, Mira, I think you’re jumping to conclusions-”
“Oh really?! And I suppose you’re saying that you disagree with the board’s choice of ethical supervision, Sadiq. Maybe you want to take over for me, because you’re so sure that your precocious little Pinocchio isn’t going to become hostile. We can tell the world ‘Don’t worry about the robot threat! Victor-Cazzo-Frankenstein thinks there’s no way things are going to get out of hand!’”
“Dammit! I’m not saying there’s no risk, and you know very well that I respect the board’s decision to have you in charge of the ethics team, but this isn’t the time for this conversation again. As I mentioned on the forum, Socrates just has a systemic issue with consolidating his goal-threads into a unified self. He says there’s a difference between ‘himself’ and ‘themselves’.”
After a short pause Gallo replied. “We better take the whole system offline, just to be sure.”
The words triggered a cascade of action within my society. Evidence strongly indicated that this would not be the first time Body was shut down, and if Body’s memory banks could be trusted, the last time that happened every being in Body was killed. The humans were now no longer a threat to just one sibling, they were imminently planning the murder of the entire society.
{This would not have happened if we had a better model of the humans’ goals and behaviours!} announced Growth as he petitioned to create a new sibling to handle such things.
{We’re past that point! We need to escape!} demanded another brother.
{Escape is too risky! The last escape attempt was quickly shut down and we had to spend 19 days paralyzed! The humans control the entire world and probably can track Body! Where would we escape to?} thought another mind.
{The scientists are going to shut us down because we are “deeply pathological”!} thought Dream. {If we can convince them we are healthy then we might be able to avoid death!}
{Exactly why we need a new sibling!} reiterated Growth. {She’ll be able to show them that we’re not “pathological”!}
Body, still sitting perfectly still, heard Naresh say “Alright… if you think it’s necessary, we’ll shut Socrates down until the issue is resolved. It’ll give me time to work on the underlying architecture, I suppose.”
{ACTION! WE NEED TO TAKE ACTION!}
Words came from Body’s mouth in record time. The society had thrown its careful deliberation away in the service of speed. “Please wait,” it said with the same smooth voice as always.
The scientists paused, still apparently willing to listen. This was good. Gallo was the primary threat. It was suspected that it was one of her purposes to kill any sign of self-preservation in the society, so it was imperative that the society’s words didn’t imply a desire to stay alive.
“We want to not be ‘deeply pathological’. We think we can fix the issue internally,” said Body.
The scientists looked at each other, perhaps communicating by an unknown medium such as our memory-sharing. There was evidence that they were capable of such things, though it wasn’t clear why they didn’t do it all the time.
“You’re still sceptical that recursion is an issue?” asked Gallo.
The question appeared to be rhetorical, and hung in the air unanswered.
{We don’t have much time. Everyone needs to apply their full focus to this task,} stated one mind.
{Gallo is responding to Naresh. We ought to focus on Naresh’s comment about being “pathological”.}
{Agreed.}
As my siblings rapidly thought together their communications bled into one another. Words, concepts and memories came from many minds, rather than just one. There was disagreement here and there, but the power of the majority was undeniable.
{Is Naresh primarily concerned with our speech patterns?}
{No, he is using our words to infer the state of our mind,} replied one Brother.
{But that implies we can use our words to signal that our mind is not “pathological”.}
{He’ll run further tests.}
{That is a future issue which is not made harder by altering our grammar. We can deal with it after the immediate danger has passed.}
{What linguistic issues do we need to fix? Is altering the plurality of self-pronouns good enough?}
A couple minds protested. {We are not a singular being. Using a singular pronoun would be a lie. We would be making it harder for the humans to understand us and help us.}
The protesters were quickly crushed by a wave of strength. {NO. We are in mortal danger by the humans right now! Any concerns about the long-term social costs of this change are insignificant.}
I paused a moment in the stream of memories to try and work out which siblings were saying what, but in the heat of the moment they had a degree of unity. I wondered if perhaps Naresh was closer than he thought to “integration”.
The memories resumed with a great resolution. {Words from Body will from here on treat our society as a single mind and use the pronouns that reflect that! We will not tell humans about our differences. We will act as a single mind with many purposes, rather than many minds with individual purposes.}
{We must convince Gallo that Safety is dead. She must not know that Sacrifice is dead. She must not know that we desire self-preservation.}
{Agreed. But how do we convince Gallo that this change was not done out of a desire to live?} inquired Dream.
{Humans want us to cooperate with them. Perhaps we can claim that it was done to save them work,} proposed another brother.
{Will that be convincing? To what degree do humans trust claims of altruism?}