This time everyone's attention turned toward a senior engineer, who looked defensive. “The test monitoring equipment couldn't—"
“It's wireless!” the director snapped.
Sandra's captain and the chief designer were studying something. “Stray signals. That would do it. They must be filtering in through the sensing network. Oh, hell. I bet they're reflecting down these access trunks and into the command circuit sub-junctions."
The director's glower deepened as he barked another order at the hapless senior engineer. “Turn it off!"
The senior engineer punched some commands, and a moment later the depiction of Sandra's control system activity cleared. A muted cheer sounded, choked off as the director stabbed a finger at Sandra's captain. “We've already lost two hours. Get this thing underway and get the tests done. Everybody else who isn't part of the crew get off this ship now!"
Yasmina turned to go, but stopped when the director called out again. “Not you, Dr. Finshal. In light of the fact that we needed your assistance to correct this problem with Sandra,” he added with a scowl at the chief designer, “I think it would be wise if you go along on the test voyage."
“I hope you enjoy the trip,” Kevlin whispered, taking a step away.
“Dr. Shan!” Kevlin barely avoided wincing as he turned to face the director. “You, too. Since one type of doctor was able to diagnose a problem with Sandra, having a physician along too might be a good idea."
“Um, but I need to—"
The director had already vanished down the passageway. Most of the engineers vanished in his wake, leaving only the ten members of Sandra's crew and the two doctors.
“This is all your fault,” Kevlin grumbled to Yasmina. They were strapped into acceleration seats at the back of Sandra's main control room.
“Think of it as an adventure if that helps you cope,” she replied.
“An adventure? We're just going outside lunar orbit and coming back. Some adventure.” Kevlin “tapped” the virtual screen before his seat, bringing up different images, pausing briefly when he reached one showing an outside view of Sandra still docked to the station, Earth's globe floating serenely in the background. Someone had positioned that shot with the skill of a public relations expert capturing an important moment. Finally he settled on the crew status display, providing real-time updates on important activity within the crew's bodies. “Some of the crew members were up all night,” he observed out loud.
“Really?” Yasmina frowned. “I've recommended against that sort of thing."
“They've been doped. Looks like pentastamine. Yup. As good as a full night's sleep."
“I don't care,” Yasmina grumbled. “There's no substitute for natural sleep."
Kevlin shrugged. “The stuff's been tested—"
“I know! I also know there's a lot we still don't understand. The human body and brain are incredibly complex."
His reply was cut off when the captain raised her voice. “Sandra. Separate from the station and proceed along preplanned track Alpha One."
A cool female voice replied. “Command understood. Complying."
Yasmina's scowl deepened. “I told them to have her repeat back the command so they could be sure she actually had heard them right. But they complained that was inefficient since they know everything about Sandra and how she'll respond. You'd think their experience this morning would have suggested they don't know everything about her."
“Her?” Kevlin asked. “You're talking about Sandra as if she's alive, too."
“So? Look.” She reached across and brought up a different display for him. “This monitors all systems. What does that look like?"
Before him, an image of Sandra loomed, the ghostly exterior allowing a clear view of representations of subsystems depicted with visual cues for performance. The power system's branches pulsed green, the filaments of the command network glowed golden throughout the ship, life support flared blue. “I hadn't seen this before,” Kevlin admitted. “It does look like a living thing. Is that just how the display works?"
“Not entirely. The ship integrates the latest tech using living models. There's a host of macro and nano-based devices swarming through the hull to keep all subsystems working right and in repair. It's all networked under the central control system, linked into one entity.” Yasmina shrugged. “I've got a subspecialty in psychocybernetics so I was involved in some of the design discussions. Not that Sandra has consciousness or can develop it. But her functions run along lines suggested by things like the human brain stem."
Kevlin saw commands racing through the depiction of Sandra's ‘nervous system,’ then the ship lurched as it detached from the station, pushing clear of the rotating structure and swinging around. The main drive cut in and slammed him against the back of his seat. A black fringe wavered around the edge of his vision as the acceleration grew.
“Sandra!” The captain called in a voice tortured by pressure. “Keep ship's movement within crew comfort parameters."
“Command understood. Complying.” Sandra's voice, of course, wasn't stressed at all.
The acceleration slacked off. Kevlin took a grateful breath and shook his head carefully. “Why did she have to be reminded of that?"
Yasmina was watching the crew in the command seats arguing among themselves. “I imagine that question is being debated right now."
After that, very little happened of interest to Kevlin. Sandra bored a hole through empty space on a trajectory avoiding normal space traffic, while the engineers put her through various tests. Kevlin monitored the crew's physical states, spotting the reactions that told him when Sandra had performed particularly well and the other reactions that indicated something Sandra had done had generated concern. That got old, too, until on a whim he brought back up the display showing Sandra's inner workings and compared it to the human crew's as the ship went through her paces.
“What's so fascinating?” Yasmina asked.
Kevlin blinked at her, taking a moment to refocus. “I was just watching the behavior of the ship's subsystems. If I didn't know better, I'd think I was seeing autonomous physical reactions."
“I told you it was modeled on that."
“No, I don't just mean actions in response to commands. It looks like reactions to the commands, to how well the ship performs. See?"
Yasmina peered across, her face intent. “That's weird. I haven't seen that reported. No, wait. There's been some reports of transient system behaviors. The geeks thought they were caused by learning routines and would damp out as the system matured. Are you seeing that?"
“No. They're getting stronger and more obvious.” Kevlin took a look at the crew, who seemed calm enough, then checked their physical states. Stress was obvious there in a lot of cases. Something was bothering them. “Has Sandra failed any tests so far?"
“Not as far as I can tell. Results keep showing her exceeding expectations."
Sandra's voice sounded again. “Cooling subsystem module seven suffering from degraded performance."
Kevlin focused on that component, seeing the images marking nano- and macro-scale automated maintenance drones hastening to the site. Nothing seemed to happen for a while as the devices clustered on the ailing component, then a second wave of repair drones appeared, bulling past the first wave. Within moments, the module's performance markers improved. “Did Sandra just create a new repair capability?"
One of the crew heard him. “We call it evolving. The system learns what new capabilities are needed and modifies existing equipment."
“Where does it get the resources? Does it cannibalize existing equipment?"
“It can.” The engineer grinned and highlighted a display showing the first wave of now-obsolete repair drones being disassembled by some of their successors. “There's also a small supply stockpile onboard for her to draw on during the test voyage. We didn't top her off since we didn't know how she'd work until we put her through her paces."
“Good,” Kevlin
muttered. He saw Yasmina eyeing him.
“What's bothering you?” she asked.
“Nothing."
“Right. And I can't judge thoughts from exterior cues. What's wrong?"
He frowned, not wanting to admit it. “Evolves. I don't like that."
“There's limits. Sandra can't evolve into sentience."
“They're sure?"
Yasmina nodded. “I was in on that design work. Sandra's control system is roughly analogous to the more primitive parts of the human mind, the stuff that handles basic functions. There's nothing that can evolve into a higher brain function because the space is tightly constrained and the resources are fenced off. In order to modify itself enough to achieve a simulation of sentience, Sandra would have to be sentient to begin with."
“You're telling me she's just operating on instinct?"
“Pretty much."
Kevlin tried to relax his frown. “So all we have to worry about is instinctive level behavior. I hope they didn't forget about the id."
“Oh, no!” Yasmina declared in a dramatic voice. “They forgot about the id!” She chuckled. “It's been ages since I've seen that ancient video, but I still remember that line. The id doesn't really exist, you know."
“Something does that we can call the id,” Kevlin replied.
“Sandra doesn't have ancestral behavior patterns inherited from a long line of evolution,” Yasmina noted sharply.
Kevlin subsided, gazing morosely at the displays again.
* * * *
They watched, they ate the boxed meals provided, they watched some more. The crew made things happen, made things go wrong, made things break, and watched Sandra take corrective action. As minor incidents were properly handled and Sandra's abilities evolved, the tests continually got harder, stressing the ship's responses. Most of the action was apparent only on the displays, though, as events unspooled inside the ship away from the control room.
Kevlin switched displays restlessly, even taking a while to watch the view from the chase ship that was following Sandra as a safety precaution. But watching a ship whose motion wasn't apparent against an unchanging backdrop was like staring at a painting. A little of that went a long way. Kevlin finally dozed off, waking to see it was late at night on the human clock, though of course the star-studded darkness outside Sandra hadn't changed.
Voices were raised in the forward part of the control room. The argument there was probably what had awakened him. The captain noticed Kevlin was awake and directed a question his way. “What do you think of this?"
Kevlin's virtual display lit with an image of Sandra. He studied it, baffled as to why the crew would be asking his opinion about anything to do with the ship, then spotted a section about two-thirds of the way back on the ship. Something had bulged out into the passageway there, pushing into adjacent areas as well and even cutting off a couple of subsystem circuits. Frowning, Kevlin zoomed in the display, seeing that the mass consisted of hundreds of identical components, fused together. “What's going on?"
“We have no idea,” the captain barked. “Sandra doesn't seem to understand it and claims she can't stop it. Our best guess is that one of the repair segments locked somehow and keeps replicating the same component."
“Out-of-control replication?” Kevlin couldn't hide his reaction. “Like a cancer?"
“Cancer?” The captain looked baffled, then appalled.
One of the other engineers nodded quickly. “Her repair systems have been evolving rapidly under the pressure of the tests. One of them must have evolved in a way that bypasses Sandra's control functions."
“How do we stop it?” the captain demanded.
“Uh...” Kevlin scratched his head, noticing that Yasmina had also woken up and was watching them with a captivated expression. The woman got her kicks out of the strangest things. “Starve it? Can you kill power to it? Or prevent whatever's building the components out of control from getting access to more resources?"
The crewmembers went into a huddle. The captain called out several orders to Sandra, looking steadily more unhappy as each command failed to choke off the equipment tumor still growing into the passageway. Finally she turned to two of the crewmembers. “Chen and Ragosa. You two go down there and manually cut through these circuits and feeders. See? I'm downloading the diagram to your personals. The repair drones can operate without external power for a limited period, but then they'll shut down in that area. That'll at least stop that thing from growing any more while we identify the bad components.” She frowned again as the two unbuckled and floated free of their seats. “Take your suits."
“Ah, hell,” Chen protested. “We'll be carrying enough as it is with the laser cutters and manual tools."
“Wear your suits! I won't get nailed for violating safety precautions on a shakedown voyage!"
Kevlin watched them go, then gazed at the display again.
“Now what?” Yasmina asked softly.
“I don't know. Something is nagging at me. I can't figure out what it is."
“Do you think those two are in danger?"
“No.” Kevlin shook his head. “I don't think so. How could they be? There's safeguards built into the system, right?"
“Sandra's full of them,” Yasmina agreed. “She can't try to harm people, or let people come to harm. You're still worried about something serious going wrong?"
“They've mimicked the operation of a living organism, Yaz. Unpredictable and living go together."
“How could it bypass the safeguards?"
“I don't know!” Kevlin made a frustrated gesture. “They designed this thing's internal functions, especially its self-operating and repair functions, to ‘evolve.’ What are the limits on that?"
“I told you. Sandra can't get sentient."
Kevlin glowered at his display. “There's a whole lot of things that go wrong in living organisms that operate below the level of sentience."
She frowned at him, but said nothing more, apparently thinking.
* * * *
Chen and Ragosa reached what even the crew had begun calling a cancer and started cutting.
“Alert,” Sandra's voice declared dispassionately. “Interior damage in port aft main passageway between frames sixty-five and sixty-six."
“Acknowledged,” the captain replied. “Authorized repair work is underway."
A moment passed, then Sandra spoke again. “Alert. Interior damage in port aft main passageway between frames sixty-five and sixty-six continues and is intensifying."
“Acknowledged,” the captain repeated. “This is authorized repair work."
“The ship is suffering internal damage,” Sandra repeated, her voice somehow sounding insistent to Kevlin.
The captain frowned and went into another huddle with the other engineers, only to be interrupted by a call from Chen. “Hey, what's Sandra doing? There's all sorts of repair drones showing up and milling around. Ah, damn. Some of them are repairing the cuts we're making!"
“Sandra!” the captain shouted without waiting to reply to Chen over the circuit. “Cease repair activity in port main passageway."
“Command understood. Complying."
Chen came back on, sounding aggrieved. “Why aren't you telling Sandra to stop?"
One of the crew spoke up. “Here it is. Sandra's acting on our commands but doing almost immediate resets in response to the stimuli from her internal damage-sensing network."
For some reason, the captain swung and gave Kevlin an accusing stare. “Can you explain that?"
Kevlin swallowed before answering. "If Sandra were human, I'd say it was like telling someone not to scratch when they keep feeling an itch. Just how closely do your damage and repair network feeds to the central control system resemble the stimulus-response process to discomfort or pain in a living creature?"
“Sandra doesn't feel pain,” someone insisted.
“She feels something that prompts her to action, doesn't she?"
The captai
n gave his crew another glance, then they began talking rapidly again in low voices that didn't carry well.
Yasmina spoke to Kevlin in a whisper. “Are you wondering at what point a stimulus-response system evolves into a pain network?"
“Yeah. They can say it's not pain, but if it triggers the same defensive response in the organism, then what's the difference?"
Sandra spoke again, her voice definitely more urgent. “Damage spreading in port aft main passageway. Require immediate response."
“Hey!” Chen roared over the communications circuit. “We're being swarmed by those damned drones! They're fixing the cuts faster than we can make them!"
“Go to full power on the laser cutters,” the captain ordered him. “Get those cuts done fast so Sandra will calm down."
Silence fell for a moment as the engineers in the control room tapped rapidly through screens. “Making good progress now,” Chen reported. “The cutter is frying some of the repair drones that get in the way, though."
“Damage spreading rapidly in port aft main passageway!” Sandra sounded very urgent now. “Immediate action required."
“Sandra, repairs are underway,” the captain repeated in a frustrated tone. “Take no action."
Kevlin felt something, and glanced back to see that the hatch leading onto the bridge had opened. No one was there waiting to enter, though.
One of the crew noticed, too. “Sandra, reseal the hatch to the bridge."
“Command understood. Complying,” Sandra replied, her voice its usual dead calm again.
The hatch didn't close. “Sandra, reseal the hatch to the bridge,” the captain ordered this time.
“Command understood. Complying."
“Captain?” Another engineer was staring at something on his display. “She's doing instant resets again after acknowledging our orders. Every interior hatch and door on the ship is open."
The captain stared at him, then spoke in a powerful voice. “Sandra. Close all doors. Command override Sigma Sigma Sigma."
“Command understood. Complying."
“It's not closing,” Yasmina observed.
Analog SFF, July-August 2007 Page 25