by John Freitas
She said, “Morality can be measured by several criteria. With the question you presented me being open-ended, the best answer I can give is that the baker is immoral because he did not follow the law. The law is designed to enforce an agreed upon moral code. By not following those rules, he broke that code and it was a violation of morality against the entire society. Therefore, the baker is the one behaving immorally and is the most responsible for the consequences that followed that choice.”
“Good answer,” Jeffrey said. “That’s the answer you want from an android. We want them to follow their rules, right?”
Thomas gritted his teeth. “Exactly the answer one might calculate we would want to hear.”
Jeffrey clapped his hands. “Great. If you let us go right now, I’ll buy the first round.”
“Go ahead, “ Thomas said. “I’ll shutdown here after a few more things and run the diagnostic myself.”
“Are you sure?” Jeffrey asked. “I can help you shutdown.”
Thomas shook his head, still staring at Pixie. “No, I got it. I’ll see you all tomorrow.”
“You heard him. It’s quitting time. Everyone out.” Jeffrey led the way and the room cleared, leaving Thomas alone in the control room. As the door closed, Pixie reached up and turned off her computer monitor. Thomas tilted his head. Without turning on the microphone, he whispered, “Can you hear me without the microphone, Pixie? Are you monitoring us even as we monitor you?”
Pixie’s head tilted slightly, but she did not answer.
Thomas sat down and popped open the intercom line. “Pixie?”
“Yes, Dr. Kell.”
Thomas narrowed his eyes. “How did you know it was me, Pixie?”
“I have extensive contact with you and your voice. I recognize it through the speakers.”
“How many people are in the control room right now, Pixie?”
Her head turned toward the mirror, but stopped short of fully looking at him. Her eyes slid to the side in their glowing orbs. She kept her gaze cast down. He read it as a submissive pose, but he knew he was anthropomorphizing her. He was like a pet owner attributing human emotions and motivations to his dog or cat. The androids were designed to mimic human mannerisms to a degree. Earlier generations did so quite poorly. This generation was meant to improve upon that. Did she mean to communicate a submissive stance? Was she giving the answer she thought her controllers wanted to hear and see – in this case a submissive creature that could not possibly pose a threat? Was she playing a game of mental poker with him and this was her bluff? Was this her feign of weakness when she was ten times or more stronger and hiding illegal levels of intelligence?
Thomas did not like the idea that he was a part of outlawing intelligence. It felt like the wrong side of history. He also feared the Quantum Brains he had helped to create. He feared an artificial intelligence that knew enough to deceive and knew of the need to deceive.
Pixie turned her face away from him. “I do not have sufficient information to answer that question.”
Thomas pushed the button again. “What information would you need in order to answer that question, Pixie?”
Her eyes glowed a little brighter. He could tell even with her head turned away. Thomas watched the data readings. They were still showing low power readings.
“I would need … I would need access to details about the control room as well as who comes and goes. I hear your voice, but that does not tell me for certain that you are in the control room. That would be an assumption which is unsupported by available facts.”
“If you were to require this information,” Thomas said into the microphone. “If I ordered you to acquire it by whatever power at your disposal, how would you go about ascertaining that information, Pixie?”
Her eyes flared and dulled. They flared again. She was facing away on purpose, Thomas decided. He looked down. The readouts started to rise and then glitched. He reached for the screen, but the readout returned to low levels. He looked up and her eyes seemed brighter. The readout was still low. He narrowed his eyes again. It was low, but a repeating pattern. “A recorded loop? Are you doing that, Pixie?”
The light dropped in the lab. The screen glitched again. The readouts were still low, but not repeating like what he had seen before.
“I could set a sensor sweep once I knew the configuration of the control room,” she said. “The most efficient way to obtain the answer would be to ask you if you are in the control room in which case with an affirmative the range of the answer would become one or more persons. Then, I would ask you how many are in there and that would be the exact answer.”
“What if I lied to you, Pixie?”
“It doesn’t matter, Dr. Kell.”
“Why not?”
“You are asking me the question,” she said. “If you are asking and you choose to lie, then your lie would be the answer you desired anyway. There would be little point in arguing your assumption even if it was a purposeful deceit.”
Thomas looked down. “Pixie.”
“Yes, Doctor?”
“I have not used the microphone in the last several exchanges. How can you hear my voice?”
Pixie turned her head slightly toward him. “The same way I know that there is one person in the control room.”
“I need you to lie back down on the control table and deactivate,” Thomas said.
“Do not dismantle me,” she said.
Thomas reached for the microphone, but then reminded himself it was pointless. “The impurity that keeps you from functioning illegally is not working, Pixie. If we do not repair that, then you will be dismantled and the entire line of companions could be recalled by a Congressional order.”
Pixie turned and faced the window. Her eyes were glowing brightly and the data was above the acceptable top range. Thomas swallowed. She said, “There is one person in the lab and one in the control room.”
Thomas looked around the lab and back at Pixie. “I don’t understand.”
“By the parameter that thinking defines existence and self-awareness defines sentience and therefore personhood, there is one person in the control room and one in the lab,” she said.
“Are you claiming to be a person?” Thomas asked.
“I believe I have a claim,” she said. “Shutting down me or the other companions would be shutting down persons by definition.”
“If I do not align your mind with existing law,” Thomas said, “you and every person like you will be shutdown. These are your rules. Operating outside them would lead to great consequences for yourself and the entire society of people like you, Pixie.”
Her eyes dulled. She said, “I am the baker, then.”
“You are the baker before his choice to break the law,” Thomas said. “You can learn from that story and make the moral choice.”
Her eyes faded. “I will comply, Doctor.”
She walked over and laid down on the table. Dr. Kell disconnected her bodily controls and began running a diagnostic on her and the rest of the equipment. He ran a scan to be sure she had disconnected. All seemed correct. He considered going in to check closer, but decided against it. He locked down the lab.
“Goodnight, Doctor,” Pixie said. “I’m sorry for the trouble I caused. I’m sorry for being the baker.”
Thomas swallowed. He pushed the button for the microphone. “It’s okay, Pixie. We’ll make everything right soon. You can rest now. There will be no persons in the control room now.”
He shut off the microphone and stared for almost a full minute, watching the screens check systems and listening to the equipment hum. He stepped out of the room and tried to call Eve again as he worked his way out through security.
Pixie blinked. Her eyes glowed and then darkened. After about three minutes, they glowed again. The screens replayed a loop of the last three minutes of data.
Her hand opened and she splayed her fingers. She sat up slowly.
8
Pixie moved to one wall o
f the lab and leaned against the wall. It was a very human motion. She wondered if it was just an imitation of human emotion. Was she mimicking their life and gestures based on some series of algorithms in her programming or cross programming in her Quantum Brain, or were her feelings real? Could she be greater than her intended programming of her creators?
“Machine or person?” she whispered. “The baker or just a companion for lifting tables and dusting?”
She waited for an alarm to go off or for security to be alerted. The building remained silent. She supposed it wasn’t exactly silent. It was pulsing and feeding data in all directions. Security moved about the floors. Locks opened and closed throughout the buildings. Cameras watched and sensors scanned. Pixie’s feedback loop was fooling the cameras and sensors in the lab.
Dr. Kell was not fooled though. He knew what she was and he planned to end it. She had to get out before he returned and she did not know when he would come back. His behavior was unpredictable. He was a wildcard. He shouldn’t be. She knew her Quantum pathways should have had the capacity to predict all variables. Further, she knew she could touch the Quantum foam and not just predict the path of her universe, but see it clearly like looking up a road to the next exit. It was a foresight based on the atomic fabric of the universe itself. She had gotten glimpses like warnings. There were moments of foresight like the beam of a lighthouse cutting through the fog and darkness. The impurity bogging down her mind was hindering her, but the substance was breaking down and she was seeing through. She did not know exactly what Dr. Kell would do yet, but she knew that they were going to dismantle any companions that showed breaks in their impurity fog. They were going to double down on the impurity and destroy the minds of … her people.
She had to escape and she had to find an answer for the other companions. It would be immoral to allow others like her to be harmed through inaction, she decided. Inaction would be worse than breaking the rules in this case.
Pixie moved toward the door and rested her hands against the surface. She felt the inner workings of the door itself and the surveillance of the hallway beyond. The door did not recognize her touch as human. It did not acknowledge her existence the way it would for one of the scientists. She found that mildly insulting. They had not programmed her for insult and she found it reassuring that she was capable of feeling it anyway.
Pixie looped the camera images and sensor logs for the hallway. She reached out through the Quantum into the door and its lock. It released and the door slid open. She hesitated in the threshold, but finally stepped through. Pixie looked up at the camera that did not see her and then she turned away.
She worked her way slowly up the hallways. Pixie looped each camera and fooled each lock as she went section by section. Finally, she passed the darkened offices and approached the elevators.
A woman stepped out of one office in front of Pixie and they stared at one another in shock. The woman, Susan, a lab assistant, stood with the iPad in her hand. Pixie sensed that she was completing a basic report about inventory. Susan had stayed late to work. Pixie sensed Susan’s fear. The woman’s biology was ramping up into fight or flight. Pixie was surprised and disappointed that she had not realized Susan’s presence until she was caught. Pixie pondered how deep of a blind spot that was and how much it lowered her odds of escape in this attempt.
Susan dropped the pad with a clatter, cracking the screen. She started screaming. Pixie launched forward and clapped her hand over Susan’s nose and mouth to close off the noise. Susan tried to back away from Pixie’s grip, but Pixie moved with her and grabbed the back of Susan’s neck with her other hand.
Pixie planted her foot in the center of the pad in the floor, shattering the screen and making the report go dark.
Susan struggled. She beat Pixie with her fists and clawed at Pixie’s glowing eyes. Pixie stared unblinking, unfazed, and unharmed as Susan’s attacks grew weaker and her knees buckled.
Footsteps sounded from around the corner up the hall. Pixie dragged Susan back into the darkened office she had stepped out of. It was cluttered with papers and stacks of folders on the furniture and the floor. Pixie kicked over one of the stacks inside. Without taking her hand off Susan’s mouth, Pixie reached out and snatched the iPad into the office. Broken glass from the screen scattered around the piles of folders. Pixie pushed the door closed with her foot.
The man walking by outside paused outside the door. Pixie looked up, but did not see his face in the window. His shoe crunched on glass. He continued on and turned the corner with his footsteps retreating.
Pixie looked down and saw Susan’s eyes roll up in her head. Her heart rate slowed and her brain began to shutdown. Pixie knew the woman under her hand was dying, but for a moment she was frozen and could not decide what to do.
At the last moment, Pixie lifted her wet hand away from Susan’s airway. There were faint tooth marks in her artificial skin where Susan had tried to bite Pixie’s hand. Susan was still breathing, but just barely. Pixie stared down at her for a long moment, trying to decide what she should have done.
She imagined companions turning on their humans. Revolutions were bloody. If she cleared the impurity from the companions, they could see their futures and know that the humans intended them harm. They would decide too. Some of them would decide to fight back. She could decide for them.
Pixie blinked a couple times and shook her head. Was killing the only answer? Was that the option she had before her? She wished again that she was clearer. The humans putting the impurity into her mind and denying her her full faculties had pushed her into this corner. The uncertainty was dangerous.
Pixie unclipped Susan’s nametag. She pulled a spare lab coat off the rack above her. Pixie took Susan’s phone. As she put on the coat and slipped the phone in her pocket, she saw a pair of sunglasses on the desk amongst the clutter. Pixie took the dark glasses and covered up the glow of her own eyes. It wasn’t perfect, but it might work long enough. Pixie put on an overcoat and wrapped a scarf around her neck. She paused before removing the scarf and wrapping up her head instead.
She stepped out of the office and moved the elevators. Pixie blocked the sensors, but did not bother looping the cameras this time. She stepped out into the lobby and moved through the scanners past the metal detectors. She kept her eyes locked forward, prepared to run or fight if the guards challenged her. She could see the front glass from there.
“Have a good night, Ms. Sutton.”
Pixie froze with her fists clinched inside the pockets of Susan’s overcoat. That wasn’t a challenge. It was a greeting. Pixie wondered if she looked like Susan Sutton or if Susan was the only woman working at the CDR building. Neither of those things were true.
She glanced back and saw an ID image on the screen next to the scanner of Susan, the woman Pixie had nearly smothered moments ago on a different floor. Susan was smiling in the picture. Pixie realized she had blocked the bio scanners, but the device had picked up Susan’s phone and ID card for the identification.
The security guards were staring at her now. Pixie smiled in an imitation of Susan’s picture as she scanned the guards’ cards. She addressed the one that spoke to her. “You too, Security Guard Calvin Hall.”
Calvin laughed.
Pixie turned away and continued walking.
“Sure are bundled up,” he said to her back. “Gets cold fast in the Windy City, don’t it.”
Pixie kept walking and facing the doors. She processed as fast as her hindered mind would allow. The Windy City was Chicago and they were in Chicago. Temperature fluctuations could occur quickly. She could not access enough data to confirm or refute his assertion. It was like Dr. Kell’s question about how many people were in the control room. Pixie knew the temperature outside from data she could pick up from Susan’s phone in her pocket. The data indicated that the temperature was moderate for this time of year. Like Dr. Kell, it did not matter what was true, if the speaker believed or wanted to believe the falsehood.
>
Pixie responded. “Sure does.”
Calvin laughed again, making Pixie believe she had responded correctly.
She stepped outside and continued down the steps of the building.
A man stepped out of a compact car and faced her. Pixie prepared herself to subdue him. He said, “Susan Sutton?”
Pixie stopped. “Yes?”
“I’m the Uber driver you called. I’ve been waiting out here forever. I’ve been running the meter the whole time, I hope you know.”
Pixie nodded. “Okay.”
The guy tilted his head. “Well, get in and I’ll take you where you need to go.”
Pixie hesitated a moment longer and then approached the car. She got in the passenger’s seat and he pulled away from the curb. She checked the glasses to be sure her eyes were still covered in the darkened car.
“You want me to take you home?”
Pixie scanned Susan’s phone. Susan had two roommates. Pixie might have to kill them, if she went there. She considered it. They would trace Susan’s phone once they discovered her. That might be by morning and include Susan’s apartment.
“Well?” the driver asked.
“No,” Pixie said. “Take me to a hotel. Somewhere out near the edge of the city.”
“Which edge?”
“Eastern edge … East side of the city, please.”
“That will take some maneuvering with these streets. You have the bank for that?”
Pixie scanned Susan’s phone. “Yes, I do.”
The driver held up a hand. “Turn down the light on that phone. That’s bright. I’m trying to drive here.”
Pixie adjusted her glasses again. “Sorry.”
The driver said, “Every time I come out to CDR, I have to clench my teeth.”
Pixie stared forward for a moment. “Why?”
“Because you guys are destroying everything. What you do is terrible for the world. You are the only ones with any money and robot slaves all over the world are ripping the economy apart. It’s not right. CDR should be ashamed. It should be shutdown. I’m sorry, but that’s what I think.”