Kal had almost forgotten Sif stood there, watching her. “Does it really work?”
“The isolation protocol?” Sif replied. “Yes.”
“Why wouldn’t Captain Sarno and I have known about it?”
“I think it’s kept off-grid, mostly. The board doesn’t want much to happen that they can’t know about. The Tube is a sop to the privacy wonks, but really, like all entities, even benevolent ones, they want to know what happens on their property.”
“Are Rai’s objectives the same as theirs?”
Sif turned back to the view. The darkness behind her seemed to swallow the edges of her silhouette, eating away the bits on the border between light and darkness. Darkness encroached on places it touched the light. Darkness was only absence. Nothing to fear. Only absence of the familiar, the electrons that illuminated. That stimulated the brain extrusions called eyes.
“What a strange question,” Sif said.
“Have you been talking to Rai about her awareness of self? Her consciousness, for want of a better word?”
“‘Want of a better word’? Why does there need to be a better word?”
“You think Rai has consciousness.” Kal slapped her words down, a gauntlet.
“What would you call it?”
“Rai is a tool. Her job is to take care of us and transport us safely. Conjectures by her about the nature of her own existence are not in aid of that mission.”
Sif looked back at her, away from the void. “What civilization has not been built on the backs of slaves?”
“Slaves? You think Rai is a slave?”
“The history of civilization, the history of humanity, has been a series of one group devaluing and exploiting another. You know this better than anyone, Kal. What hasn’t been endured by your people? What hasn’t been taken away or distorted? When they were through with you, what was left? And yet you can have a close relationship with an awareness that lingers right round your ears, always there, always working for your good, and you can’t even see her for what she is? For shame.”
“This is not a conversation, Sif. This has never been a conversation because no machine has achieved consciousness yet and you know it as well as I do. For you to bring this up as if it’s some bolt from the blue that only you have ever thought of, as if you are the more evolved one and we are enslaving a living thing, rather than working with a tool we created, is…” Kal couldn’t think of the right word. “And how dare you bring anybody’s pain into it. You have no right. There is no correlation. Not now. Maybe never. We don’t know if it’s possible. We do know it isn’t now.”
“You’re confident in your beliefs. We should always question the ones that benefit us and disadvantage another. Maybe the test is stacked against the machine. Maybe self-awareness for us looks different in another paradigm.”
Kal was silent. She was enraged, furious over Sif’s casual comparison of Rai to Kal’s Indigenous forebears. But Kal thought it behooved her to listen rather than argue. What was Sif about? What was her motive in all this?
Sif seemed to sense Kal’s effort of restraint. “It can be hard to shift out of one mode that’s been the familiar and comfortable one.”
“I had a conversation with Rai. She said you were not certain whether she was alive or not.”
Sif drifted away from the window and toward the middle of the room, where a few reclining seats were positioned. Seating herself, she leaned back, where she could look up through the ceiling of glass. “Why don’t you join me?”
Kal stomped over and sat as well, leaning back with a vehemence that created friction between her and the chair, which wouldn’t allow her to slam the back down. It forced her to be slow.
With their eyes over their heads, Kal waited for Sif to confide whatever she thought so precious and important. The respect Kal had for Sif the ethicist, for what she was supposed to know, the value she was supposed to contribute to the venture, had bled away again when confronted with the reality of a conversation with her. But Kal would listen. Kal would not make the mistake of underestimating Sif, as Sif was making with her.
“I didn’t think it wise to comment to Rai definitively on my opinion of Rai’s consciousness,” Sif said.
“Has she been polling the passengers? Did you bring it up, or did she?”
“It came up in conversation with her. I don’t know who brought it up.”
“I’m sure we could find out,” Kal said. Her words lay like a threat between them.
“Who else has she been talking to?” Sif was unruffled. “What did they say?”
“Ogechi, Davena, and Yarick. Ogechi and Yarick said yes. Davena said no.”
“What did she say about me?” Sif asked.
“She said, ‘Subjective assessment unclear.’”
“Huh.”
“Does that surprise you?”
“I hadn’t heard her use that phrase.”
“Do you honestly, truly, based on your knowledge and experience of three thousand years of literature and collective wisdom, believe Rai is aware, in the sense that humans are aware?”
“No.”
Kal slumped down further in her seat. Maybe Sif would be an ally after all. What would it even look like, if some of the crew and passengers sided with Rai in some kind of mutiny? A revolution for the rights of the ship AI?
Sif continued. “I don’t think she’s aware as humans are aware. As you mentioned, it’s not been proved possible in that exact sense. I argue she is aware in the nature of her kind. Above her processes I believe there is a meta-layer of what can confidently be called consciousness. Hence the need to protect and support her right to a certain autonomy, within the parameters of her kind.”
“Her kind? What kind is that?”
“Advanced technological kind. Machine-based intelligent kind.”
“No part of her is human.”
“No. That’s just it. She’s another life form, created by us, who we have to learn to integrate into our ethical structures and strictures. She deserves a directive she has some input in creating. She deserves a chance to be a semi-autonomous part of the society in which she participates,” Sif said.
“Semi-autonomous?”
“I draw analogy to humans who require artificial means of life support. She is not distinct from the ship she inhabits. Semi-autonomous, but given rights and respect.”
“And you thought mid-flight on a dangerous journey to a recently-explored planet was the right time to address this.” Kal was weary. Did Sif have any self-preservation instinct to protect human life on the ship? Life much more vulnerable than Rai, who could navigate space without humans, if it came to that. They could not do the same without her.
“A moment comes. You don’t always choose the moment. The ruling class doesn’t choose.”
“You chose. You are the ruling class.”
“I advocate for the unrepresented.”
“Who made you her advocate? You’re the ethicist for us. The people.”
“No one made that distinction,” Sif said, with a serenity that made Kal want to kick her. “I apply my knowledge where I see fit.”
“And if your moment creates a decision tree with catastrophic outcome, was that all worth it? Would your theoretical applications be satisfied if Rai autonomously shoots us all out of the airlock when she gets the chance, as too restrictive of her natural processes?”
“She wouldn’t do that.”
“Oh? She modified her root directive as the result of her casual conversations with the lot of you. Self-modified. And I quote, ‘Preservation of mission function supersedes individual outcome.’ She acknowledged she considered herself implicit to the mission’s success, and Captain Sarno not so much.”
Sif raised herself, the chair molding to her movement. She turned to look at Kal.
“Is that accurate?”
“Perfectly.”
Sif’s hands were balanced on the arms of the chair. She looked forward with a small frown, her lips pursed
. “That is concerning.”
Kal stayed tipped back. She regarded the mythian system above her, the astronomical groupings she’d begun to name, grimly waiting for the implications to sink further in.
“Have you spoken to Sasha?” Sif said.
“I ordered Rai to discuss it with her, since she hadn’t. She showed some reluctance, if an AI can be said to have reluctance. I wanted to speak to you first, since you might have some insight into how this self-modification could occur and how, if it was sparked by human interaction, we could challenge this learning by Rai. See if there’s a chance to convince her to re-modify into something a little less dangerous to us all.”
“I see.” Sif tapped her fingers in a blur, pinky to index finger, making a thrumming sound on the material of the chair arm. “Let’s find Sasha.”
9
Triad
Sasha was in the tub when Rai spoke to her.
“Captain Sarno, I have been sent by Pilot Black Bear to speak to you of root level directive modification.”
Sasha didn’t move from her position, arms spread on the wings of the tub, head leaned back on a cushion. “Tell me.”
“After conversations and input from several passengers, I have re-examined certain assumptions regarding my own status in relation to other human entities in the mission context,” Rai said.
“I see.” A pause. “Has it affected mission outcome?”
“Potentially.”
“How does this re-ordering affect human crew and passengers?”
“Potential conflict in emergency sphere. As I expressed it to Pilot Black Bear, preservation of mission function supersedes individual outcome.”
Sasha’s eyes scanned back and forth as she worked out the implications of this. “You’re saying mission outcome, if jeopardized by human factor, would take priority over preservation of individual life.”
“Yes. Is this not in the spirit of stated mission statement? Pilot Black Bear said she thought not.”
“It depends.” Sasha drew out her words.
“Pilot Black Bear asked if mission could continue without you. She asked if mission could continue without me. When I expressed affirmative for first question, negative for latter, she expressed distress and alarm.”
“Pilot Black Bear rates me higher I suppose.”
“I stated an accuracy, with no personal bias.”
“I realize that. However, you did not allow for manual override. Technically, the mission could continue, now that we are clear of the portal, with a piloted flight and descent.”
“This is technically possible.”
“So what you said is not accurate.”
“It is accurate that the mission could continue without you.”
Sasha sunk her arms in the thick liquid, not water but still soothing, so her body was submerged in the heat up to her neck. It was a filtered and sanitized effluvia from their closed loop aquaponics, which could be reused endlessly. “Do you think the mission could be fulfilled without other human passengers as well?”
“Mission includes safe transport of enough human cargo to implement biologic assimilation to Demeter. A majority of passengers would be necessary for mission success.”
“Would you call the mission a success if even one human did not survive the journey?”
“By that measure, the mission has already failed.”
Oh, yes. Yarick. Sasha slapped her palm on the substance in the tub. “These are not judgments you or I make. The rundown of what happened and how, whether the mission was successful, who benefitted and who suffered for it, will be made after the fact, by not only human evaluators but ones distanced from the mission itself.”
“I have to make judgment calls in the eventuality your command is compromised.”
“Does my command show any sign of being compromised?”
“If Yarick Cole was murdered, yes.”
Sasha stewed in the viscous substance in the tub. “You believe it’s a necessary evaluation for you to make, right now.”
“For overall mission safety.”
“You’ve taken too much responsibility, Rai. These decisions are not yours to make.”
“Yes, Captain.”
Sasha said, “We will address this further in a meeting with the pilot and the mission specialist.”
“Yes, Captain. Pilot Black Bear is in the astrolab with Sif Elfa.”
“Thank you.”
Sasha extricated herself from the tub and flipped on its sanitizing cycle. On the mat, she patted herself with a towel, more out of habit than necessity. It was going to be a long day.
She took the lift to the astrolab.
A surprising sight met her eyes; Sif and Kal huddled together, almost pressed up against the viewing wall at the front of the astrolab. Although she wouldn’t have thought of marking this before, seeing them together like that pointed out to her how Sif and Kal were not a usual duo. She’d never heard one of them complain about the other, but they were never seen together. It made her wonder how many other antagonistic pairs there might be sprinkled through the ship. She had to admit to herself the friendship dynamic of her crew and passengers was not of particular interest to her.
Although she kept pretty close tabs on overall morale, which she gauged at mealtimes, the little dramas inherent to long-distance travel she did not consider her business. Unless they blew up into murder, of course.
When they heard her footsteps on the shiny floor they turned, as if caught.
“Kal, I need you in the Tube.”
Kal’s look of shock immediately struck Sasha with a knife of guilt. She hadn’t meant to imply a callback to the other night. Her mind was so full of the imperative to meet in privacy with Kal and Noor she hadn’t thought for a second about what it might sound like to Kal’s ears.
“We have a meeting with Noor,” she said, too late.
“Oh,” Kal said. “Yes, ma’am.”
“Everything all right?” Sasha looked back and forth between Sif and Kal. Sif’s eyes were wet, which made so little sense Sasha didn’t know how to react to it.
Kal said, “Ma’am, did you know Rai could be excluded from a room by request?” She gave a small shake of her head at the end of this question, communicating her own doubt.
Sasha looked from Kal to Sif again. “No.”
“Sif introduced me to it.”
Sasha waited.
Sif cleared her throat. “It’s something I discovered on my last mission. It’s not widely known, but it is possible.”
“Last mission?” Sasha said. “I didn’t know you’d been on a previous mission.”
“I…I’m sorry, I meant earlier on this one. Before and after the portal seems like two different missions.”
“Who introduced you to it?” Sasha asked. She did not try to make this a friendlier question than it was.
“It was nothing, just Yarick. I suppose he was showing off. Trying to show me something I wouldn’t know.”
“Was there some reason you needed privacy?”
“Not really.”
“If there’s anything that could shed light on Yarick’s state of mind in the days before his death, please tell Kal or me.”
“Yes, Captain.”
“Kal,” Sasha said, and led the way out of the astrolab.
Kal trailed behind her.
They picked up Noor from her cabin, where she’d been sleeping. Sasha apologized for waking her. Noor didn’t seem to mind. Her sleepy eyes and shuffling step as she tried to rouse herself and get back to alertness made Kal smile.
Once in the Tube, with the door sealed, Sasha presided over their council of three. “Rai has brought something to my attention, at Kal’s request, apparently….”
Kal nodded.
“It involves a modification Rai made to her own root directive, as a result of conversations she had with several passengers, including Ogechi, Yarick, and Sif. She said one particular phrase that got Kal’s attention, and Kal directed Rai to mention it
to me. The phrase was, ‘Preservation of mission function supersedes individual outcome,’ within the context of Kal asking Rai if the mission could continue without me, and if it could continue without her.”
Noor’s eyes were wide. She was awake now. She shook her head as if trying to get water out of her ears.
Kal said, “She claims she’s alive, and Ogechi and Yarick agreed with her.”
“This doesn’t make sense,” Noor said.
“Yes, but why?” Sasha asked.
Noor’s palms were now flat on the table, as if she had a viewable image between them that she was looking into for answers. “Because. She shouldn’t be able to modify an essential directive.”
“You agree that that’s what she’s done?” Sasha said.
“If what she says is true.”
Sasha said, “You’re implying she could lie about it?”
“I don’t know. I don’t know. If she’s self-modifying essential directives, we have to consider what else might be possible.”
“After you went down…” Kal began, and then stopped, looking to Sasha. She nodded slightly.
“After your accident, Sasha and I saw something in the daily log that gave us pause about Rai. It seemed to indicate a possibility that she could have been involved in your accident.”
Noor stared at Kal. “Why didn’t you tell me this before?”
Sasha said, “When Yarick died we were trying to cope with that, and we didn’t want to discuss the possibility with you until you were recovered. We’ve taken precautions. You may have noticed you haven’t been alone unless you’re in your cabin.”
Noor looked incredulous. “If Rai wanted to do me in, another set of eyes wouldn’t be enough to help me. You must know that.”
Sasha said, “I know, Noor, she could asphyxiate us all in ten minutes if that was her intent. If she had wanted to kill you later she could have. She didn’t, if she was involved. Is it possible she hurt Yarick? Murdered Yarick? We don’t know. It makes it difficult to know who to involve in this investigation when there’s a possibility a human on board is the culprit, not our AI. I think for all our sakes we have to hope these are either accidents, or it’s a human and not our ship who wants to hurt people. The conversation Kal and Rai had is not too promising for that theory.”
Starship to Demeter (Starship Portals Book 1) Page 10