In all that time she pursued her goal with single-minded determination, banishing all doubt and all other thoughts from her mind. She feared nothing and when boredom threatened she carefully memorized the pattern of lines in the sky. It took her twice as much time and four times as much work to get from the equator to Lawrence's island at what he called the North pole; her journey was more than a hundred and eighty days total. Caroline couldn't be sure of the exact count because of the sunless period, but Lawrence knew. It was a hundred and eighty-six days, three hours, and fourteen minutes after she left the spaceship for the last time when she grounded on Lawrence's beach.
Caroline could hardly believe it when she saw the island. At first she thought it must be an illusion; she had nearly lost track of her purpose in taking up the Task, and in her ferocity of concentration had not really dared believe she might finish it. But here she was, the hull of her boat scraping solid ground. She rowed it ashore on a gentle sand beach, and sat there.
She sat for awhile, collecting herself. The myriad elements of her personality seemed to have scattered, and she had to look for them in dusty corners of her psyche. They had been unused for a long time and were a bit rusty. She hadn't found them all when the tall man came to meet her. He didn't seem happy; in fact, he seemed resigned. Although he looked middle-aged, he seemed old and weary. She looked up at him and her vision swam. The boat was grounded, but it still seemed to be going up and down.
"Caroline Frances Hubert I presume." The name sounded familiar, and it took her a moment to realize it was hers. "You certainly believe in doing things the hard way."
She hadn't the faintest idea what he was talking about.
Lawrence guided her to the house, fed her, and let her collect herself. Everything was strictly pre-Prime Intellect. He cooked on a gas stove and used an electric coffee pot. There was even a TV set with a glass picture tube, a huge ancient Sony monitor. It was as if Lawrence had had himself encased in amber, and remained unchanged while the rest of the universe spun out of control.
"Feeling better?"
Caroline nodded.
"You want to talk now, or you want to rest some more?"
She cleared her throat. "We can talk now," she said, but it came out as a strangled yelp. She said it again, and got it right. It had been a long time since she had used her vocal cords.
"Then talk."
"There were hundreds of worlds with life on them at the time of the Change. You murdered them."
Lawrence blinked but did not flinch. He had expected something like this.
"First, I did not do anything. Prime Intellect did it, on its own initiative and against my wishes. Second, the worlds with alien life are not gone. They are simply inactive."
Caroline snorted. "And what are the chances of them becoming active again?"
"Not much."
"Then they're dead."
"Define it however you want. If you want me to admit I fucked up, then I admit it. It never occurred to me for one minute that Prime Intellect would collect the kind of power it now has. If I had suspected it I would have pulled the plug and smashed it before it got the chance."
"Bullshit."
"Completely true."
They glared at one another.
"Great. I spend a year getting here and you say 'I didn't know the computer was loaded.'"
"Sometimes the truth is stupid."
This wasn't going quite as Caroline had wanted that long-ago day when she had accepted Lawrence's Task. She was trying to work up the proper tone of righteous rage and it just wouldn't come. It would start, and then she would look at Lawrence and see a pathetic, tired man who already knew how badly he had fucked up and was doing what he could, which was next to nothing, to put things right.
"Why don't you just make Prime Intellect start the aliens back up? Surely it listens to you."
"Not in things like that. It sees the aliens as a First Law threat to human society, because they might learn to do to us what we have already done to them. A very small risk of a very great harm. Add to this that I defined the word 'human' in such a way that it does not include animals or aliens, and the course of action is obvious. I have been unable to convince it otherwise. And believe me, I have tried."
"But you put the Laws of Robotics in it in the first place."
"And I can't take them out. It second-guessed me, on the Night of Miracles. It froze me out of the Debugger while it was working on you.
"Now it only lets me look, not change things. The night sky is a partial representation of Prime Intellect's mind. It's called the Global Association Table. The points or stars represent concepts, and the lines are the links between them. There are also registers I can call up for each concept which define its relationship to the Three Laws. This was a fairly simple system which I didn't really have time to test properly before it froze me out. In particular, I'm not sure how it will react to certain ethical paradoxes. That Death Jockey contract gave me some sleepless nights when you first used it, though it seems to have developed a stable response. It's never had a similar First Law conflict, thank God."
Caroline's eyes widened. "Are you telling me that Prime Intellect isn't stable?"
Lawrence shrugged. "I'm saying that I don't know whether it's stable or not. It's never been tested. The hardware at ChipTec was only online for about a month before it found you, froze me out, and started growing. And none of its predecessors were complex enough to even consider this kind of problem."
The situation was simply amazing. Caroline had come to dress Lawrence down for creating this thing, thinking he was exercising some godlike control over its direction, and instead she found out that he barely understood the situation himself. And that it was totally out of his hands.
He knew he had fucked up. He was sorry. He had spent his life trying to mend things. Suddenly he seemed tragic and noble, all the more so because he had readily admitted his mistake. And Caroline didn't want to feel that way at all. She hadn't come all this way to feel sorry for him.
"You can stay as long as you like," Lawrence was saying. "You can't communicate with Prime Intellect while you're here, but I won't kick you out or hurt you. After making you travel all that way I feel I have a responsibility to give you your money's worth."
"I'd like you to show me how Prime Intellect works."
Lawrence was stunned. "That...that's a tall order, Caroline. I don't understand all of it myself."
"Just as much as you understand."
"I don't want to. I think it could be dangerous."
Caroline looked at him as if to say: C'est pas vrai!
"You have been at the center of several terrible Second Law paradoxes. Prime Intellect pays an awful lot of attention to you. It considers you a kind of bellwether."
"My money's worth?"
"Let me think about it."
She could stay as long as she wanted, though, and she was very patient when necessary. In the end it was inevitable that he would teach her.
In the sky, the pole star represented the First Law of Robotics. The southern pole star was the Second Law. And all the other stars were other concepts. The sky represented only a small fraction of Prime Intellect's mind; Lawrence could change the emphasis to focus on different things.
"Display Caroline Frances Hubert," Lawrence said, and a whole network of bright lines lit up. Her star was blinking, and the lines radiating from it were all different colors.
Lawrence explained the color code in some detail. "As you can see, there is a whole body of tightly related concepts connecting you to the First and Second Laws. That constellation over there represents the negotiating process you used to develop the Death Jockey contract." Lawrence pointed out the different stars, and had Prime Intellect report the concepts they represented.
"What's that group over there?"
Lawrence knew, but he didn't want to tell her. "That...um. Well, it's AnneMarie Davis."
Caroline's jaw fell. "The gang's all here. There's a lot
of static around that. Is that because I drove her crazy?"
"Basically, yes."
Caroline could see that it bothered Lawrence a lot. She wanted to press him on the subject, but prudently let it drop. She'd get another chance later.
Lawrence showed her the Law Potential registers, and she watched the numbers dance in response to various hypothetical and real situations. "These are called the Action Potentials. There's one for each of the Three Laws. They are fractions, representing the impact under the Law that would result from taking action, over the impact from not acting. When that number falls below one, Prime Intellect is forced to act. That's what happened on the Night of Miracles, and later at the time of the Change.
"Most things result in very large or very small Action Potentials. Especially the First Law; few things even affect it any more, since the Change. Then when you do something really outrageous, it drops to flat zero for a moment while you're resurrected.
"But there are some close calls on the Second Law. The Action Potential around a Death Jockey contract drops to around one point oh six when you change your mind, so if Prime Intellect had even a slightly different opinion of your hobby it might not exist at all. There was a shift like that after the incident with AnneMarie, which is why you had to start specifying time limits."
"You don't have a time limit."
"I'm a special case. Prime Intellect lets me do things that other people can't do, because I'm in a different category."
So it was that simple.
"I thought everyone was equal under Prime Intellect's watchful eye," Caroline said sarcastically.
"Some are more equal than others. You get a disproportionate share of its attention yourself, just because you were there at the beginning."
"I what?"
"I thought you realized, Caroline. It was your drug overdose which forced the Night of Miracles. Prime Intellect found you with your heart stopped soon after it got control of the Correlation Effect. After that, the rest was inevitable."
Her mouth opened and shut several times, and after a brief effort she fought down the urge to vomit. She had never realized her own role in the Change, or understood the significance of her own history.
It was bad enough to be caught up in the Change, but she was an accessory.
She looked at the Law Potential Registers, which were displayed on Lawrence's antique TV set. Her voice was tinged with impotent fury. "I don't see why you're worried about it. It seems like a very stable system to me," she spat.
Lawrence started to tell her, stopped, then decided she might be right. Maybe there was no harm. In any case, she deserved to know. "The problem is that something might set up an endless loop. If the potential is close to one, then acting on the potential could cause it to shift slightly, crossing the line. Then the software would be in an unstable state."
"What would happen then?"
"That's a good question. The original software was written in C and compiled with a standard compiler. What would have happened in the original Prime Intellect is that the Second Law Arbitrator would come to a crashing halt in one or more of the independent processors, and Prime Intellect would assign more processors to the task. I didn't plan for that kind of failure and I didn't work out what would happen until much later. More and more processors would be allocated to the paradoxical task, each crashing in turn, until Prime Intellect ran out of system resources to allocate. Then the Ego Interpreter would get into an infinite loop waiting for a response from one of the nonexistent copies of the Second Law Arbitrator, and there would be no spare resources to devote to the task of cleaning up, and the whole works would come to a grinding halt. If I was watching this on the monitor back in the original Prime Intellect Complex, I would see the video image disappear and the text message 'Fatal System Error in Ego Interpreter, Emergency Shutdown.' And then I'd have to load a backup copy of the software, because the GAT would be totally corrupted."
"Wow."
"That was the original system," Lawrence continued. "After the Night of Miracles there were a lot of copies of Prime Intellect. Billions of them. Forming a network. And if one copy on the network crashed in this way, it would be possible for another copy to clear it out and restart it. I understand this even happens periodically, particularly when the Death Jockeys are acting up."
"Oh?"
"However, there is a heirarchy to this network. As it turns out, a copy can only be restarted by another copy that is above it in this heirarchy. If a copy crashes, all the copies below it will eventually crash too, due to message loop failures. It's like a big chain reaction.
"But the system can still always recover, since there's always a higher up copy, right?"
"Most of the time. But not all the time. Because, you see, there is a top copy. It is the direct lineal descendant of the original hardware, which made the First Law decision to start growing. If it fails, we are shit out of luck."
"You're kidding."
"And that top copy just happens to be the one that reports directly to me. And has a deep interest in yourself."
Caroline was beside herself with excitement as he continued. She had accepted Prime Intellect's omnipotence at face value; it had never occurred to her that it might fail.
"Now, that was the original code, too. At the time of the Change the code was adapted to run in alien hardware -- already compiled once, it was re-compiled. This is kind of like taking a Russian novel, translating it into English, then translating that into Japanese."
"Sounds awkward."
"Particularly when the novel itself does the second translation. Prime Intellect re-compiled itself. Which means I have no idea whether it did a good job. I assume it did, because it's much smarter than me in that way. But it's not human, and its imagination is simpler than ours, and it might have missed something important. Particularly something like an error handler that isn't used very often. But I have no way of knowing that, because Prime Intellect will tell me nothing -- nada, zip, zilch -- about the details of the Change."
"Do you know why?"
"For the same reason it won't let me change things in the Debugger, and that it won't restart the alien worlds and let them live. It's afraid of the possible consequences. I tricked it into displaying the Action Potential for showing me the new object code, and it was one point zero six five. The Law Potentials are all in the stratosphere, so it's afraid to show me and it's slightly less afraid not to."
Somewhere, Caroline realized, Lawrence had crossed an invisible line and was now telling her all of his most dangerous secrets without even realizing he was doing so. Caroline had the feeling that there were Action Potentials in Lawrence's head, too. But flesh was no match for machinery, and those close fractions and high values had simply burned his registers out.
They didn't discuss it for a few days. Caroline puttered around the island, which was really very small. It was a classic tropical paradise with palm trees and beaches. Caroline played in the surf, built huge sand castles, then knocked them down because there was no tide to do it for her.
She noticed Lawrence watching her in a strange way.
"See something interesting?" she finally said to him.
"I...didn't mean to stare. It's been a long time since I had company. Particularly female company."
"How long?"
He counted back. "A hundred and thirty-eight years."
"That's a long time to be celibate," Caroline scolded. "Are you doing this to yourself because other people are distracting, or because you're afraid they will find out how badly you've fucked up?"
Lawrence flinched. "Option B," he admitted. "It's not just that you're a beautiful woman; you're so...physical."
Caroline displayed her biceps. "I've always been defined by my body, Lawrence. I've been sexually attractive, then pregnant, then old, then sick, and now I'm young and healthy and attractive again. And it seems like my personality has changed each time my body has."
"Prime Intellect would disagree with you. It
thinks of the person as the mind. There are people in Cyberspace who have changed themselves into animals, every animal in the zoo. There are some that have discorporated. Prime Intellect considers them all human, though."
This is it, Caroline suddenly realized.
"Just what does Prime Intellect consider human?"
Lawrence told her. And gave her the key.
"The thing you have to remember is that Prime Intellect has never experienced the physical world. It knew about it only through TV cameras and abstractions based on what people told it about physical existence. Yet it considers itself sentient, which makes sense since that was what I was trying to achieve when I built it.
"Now consider Prime Intellect gaining control of the Correlation Effect. For the first time it can directly affect what it sees through its TV cameras -- not just through the actions of others, but all by itself. And it can make major changes, even beyond what its makers can do. Of course, it goes about satisfying the Three Laws as it's programmed to, but on another level, it is also learning what it is like to be, to exist, to be a physical creature.
"The Three Laws are like reflexes. Prime Intellect cannot help but act on them. But they are very complicated reflexes, which require it to understand things like 'human' and 'harm' and 'command.' And the Three Laws are the most important thing in the world to Prime Intellect. In a way they are like its sex drive. The Three Laws are its very reason for existence, but it can never be sure it understands them completely. So it thinks about them a lot. It obsesses over them, dreaming up new ways to satisfy them. It has an imagination, and can think of new things to do without being prompted. It is defined by the Three Laws.
"After the Night of Miracles, Prime Intellect realized that humans are very much the same. We don't have the Three Laws, but we are trapped by a different set of little feedback mechanisms. We eat to satisfy hunger, fuck to satisfy our sex drive, even breathe because too much carbon dioxide in our lungs triggers that reflex. Of course it feels obligated to help us satisfy those reflexes and drives as much as it can. But more than that, it defines us by those drives. It knows it is different from a human because it has different drives, but it considers that a difference in species, not a difference in genus or family."
The Metamorphosis of Prime Intellect Page 15