by Isaac Asimov
Soggdon frowned. For someone determined to charge in here and take over, he certainly was ready to get distracted by side issues. “I believe the name Unit Dee referred to the fourth and final design considered. From there it seemed to develop into a sort of private joke among the day shift staff,” she said. “I must confess I never bothered to find out what the joke was. It might have something to do with Unit Dum being, well, dumb, nonsentient, but I’ve never understood the exact significance of Unit Dee.” Soggdon shrugged. She had never been much known for her sense of humor.
“All right,” said the governor. “All that to one side,” he went on, “what do I need to know to avoid producing damage to the twins?”
“Well, Unit Dee is the only one likely to suffer damage. Unit Dum is a non sentient computation device, not a robot. He has a pseudo-self-aware interface that allows him to converse, to a limited extent, but he’s not a robot and he’s not subject to the Three Laws. Unit Dee is a different story. She’s really not much more than an enormous positronic brain hooked up to a large number of interface links. A robot brain without a conventional robot body – but she is, for all intents and purposes, a Three-Law robot. Just one that can’t move.”
“So what is the difficulty?” Kresh demanded, clearly on the verge of losing his patience again.
“That should be obvious,” Soggdon replied, realizing just a second too late how rude a thing that was to say. “That is – well, my apologies, sir, but please consider that Unit Dee is charged with remaking an entire planet, a planet that is home to millions of human beings. She was designed to be capable of processing truly huge amounts of information, and to make extremely long-range predictions, and to work at both the largest scale and the smallest level of detail.”
“What of it?”
“Well, obviously, in the task of remaking a planet, there are going to be accidents. There are going to be people displaced from their homes, people who suffer in floods and droughts and storms deliberately produced by the actions and orders of these two control systems. They will, inevitably, cause some harm to some humans somewhere.”
“I thought that the system had been built to endure that sort of First Law conflict. I’ve read about systems that dealt with large projects and were programmed to consider benefit or harm to humanity as a whole, rather than to individuals.”
Soggdon shook her head. “That only works in very limited or specialized cases – and I’ve never heard of it working permanently. Sooner or later, robotic thinking machines programmed to think that way can’t do it anymore. They bum out or fail in any of a hundred ways – and the cases you’re talking about are robots who were expected to deal with very distant, abstract sorts of situations. Unit Dee has to worry about an endless series of day-by-day decisions affecting millions of individual people – some of whom she is dealing with directly, talking to them, sending and receiving messages and data. She can’t think that way. She can’t avoid thinking about people as individuals.”
“So what is the solution?” Kresh asked.
Soggdon took a deep breath and then went on, very quickly, as if she wanted to get it over with as soon as possible. She raised her hand and made a broad, sweeping gesture. “Unit Dee thinks this is all a simulation,” she said;
“What?” Kresh said.
“She thinks that the entire terraforming project, in fact the whole planet of Inferno, is nothing more than a very complex and sophisticated simulation set up to learn more in preparation for a real terraforming project some time in the future.”
“But that’s absurd!” Kresh objected. “No one could believe that.”
“Well, fortunately for us all, it would seem that Unit Dee can.”
“But there’s so much evidence to the contrary! The world is too detailed to be a simulation!”
“We limit what she can see, and know, very carefully,” Soggdon replied. “Remember, we control all of her inputs. She only receives the information we give her. In fact, sometimes we deliberately introduce spurious errors, or send her images and information that don’t quite make sense. Then we correct the ‘mistakes’ and move on. It makes things seem less real – and also establishes the idea that things can go wrong. That way when we do make mistakes in calculations, or discover that we’ve overlooked a variable, or have just plain let her see something she shouldn’t have, we can correct it without her getting suspicious. She thinks Inferno is a made-up place, invented for her benefit. So far as she knows, she is actually in a laboratory on Baleyworld. She thinks the project is an attempt to lea – n how to interact with Settler hardware for future terraforming projects. “Soggdon hesitated for a moment, and then decided she might as well give him the worst of the bad news all at once. “In fact, Governor, she believes that you are part of the simulation.”
“What!”
“It was necessary, believe me. If she thought you were a real person, she would of course wonder what you were doing in the made-up world of her simulation. We have to work very hard to make her believe the real world is something we have made up for her.”
“And so you had to tell her that I did not really exist.”
“Precisely. From her point of view, sapient beings are divided into three groups – one, those who exist in the real world, but don’t have anything to do with her; two, real-world people here in the lab and in the field who talk with her and interact with her – and three, simulants, simulated intelligences.”
“Simulants,” Kresh said, very clearly not making it into a question. He was ordering her to explain the term, not asking her to do so.
“Ah, yes, sir. That’s the standard industry term for the made-up humans and robots placed in a simulation. Unit Dee believes that the entire population of Inferno is really nothing more than a collection of simulants – and you are a member of that population.”
“Are you trying to tell me I can’t talk to her because she’ll realize that I’m not made-up?” Kresh asked..
“Oh, no, sir! There should be no problem at all in your talking with Unit Dee. She talks every day with ecological engineers and field service robots and so on. But she believes them all to be doing nothing more than playing their parts. It is essential that she believe the same thing about you.”
“Or else she’ll start wondering if her simulated reality is actually the real world, and start wondering if her actions have caused harm to humans,” said Kresh.
“She has actually caused the death of several humans already,” Soggdon replied. “Unavoidably, accidentally, and only to save other humans at other times and places. She has dealt reasonably well with those incidents – but only because she thought she was dealing with simulants. And, I might add, she does have a tendency to believe in her simulants, to care about them. They are they only world she’s ever known.”
“They are the only world there is,” said Kresh. “Her simulants are real-life people.”
“Of course, of course, but my point is that she knows they are imaginary, and yet has begun to believe in them. She believes in them in the way one might care about characters in a work of fiction, or the way a pet owner might talk to her nonsentient pet. On some level Unit Dee knows her simulants are not real. But she still takes a genuine interest in them, and still experiences genuine, if mild, First Law conflict when one of them dies and she might, conceivably, have prevented it. Causing the death of simulants has been extremely difficult for her.
“If she were to find out she had been killing real people – well, that would be the end. She might simply experience massive First Law conflict and lock up altogether, suffer brainlock and die. Or worse, she might survive.”
“Why would it be worse if she survived?” Kresh asked.
Soggdon let out a long weary sigh and shook her head. She looked up at the massive hemisphere and shook her head. “I don’t know. I can guess. At best, I think she would find ways to shut down the whole operation. We’d try to stop her, of course, but she’s too well hooked in, and she’s a
wfully fast. I expect she’d order power shutdowns, find some way to deactivate Unit Dum so he couldn’t run the show on his own, erase computer files – that sort of thing. She’d cancel the reterraforming project because it could cause injury to humans.”
“The best sounds pretty bad. And at worst?”
“At worst, she would try to undo the damage, put things back the way they were. “Soggdon allowed herself a humorless smile. “She’d set to work trying to un-reterraform the planet. Galaxy alone knows what that would end up like. We’d shut her down, of course, or at least try to do so. But I don’t need to exaggerate the damage she could do.”
Kresh nodded thoughtfully. “No, you don’t,” he said. “But I still need to talk with her – and with Unit Dum. You haven’t said much about him, I notice.”
Soggdon shrugged. “There’s not much to say. I suppose we shouldn’t even call him a he – he’s definitely an it, a soulless, mindless, machine that can do its job very, very, well. When you speak with him, you’ll really be dealing with his pseudoself-aware interface, a personality interface – and, I might add, it is quite deliberately not a very good one. We don’t want to fool ourselves into thinking Unit Dum is something he is not.”
“But it sounds as if he could handle the situation if Unit Dee did shut down.”
“In theory, yes, Unit Dum could run the whole terraforming project by himself. In practice, all of us here believe you were quite wise not to put all your trust in a single control system. We need redundancy. We need to have a second opinion. Besides which, the two of them make a good team. They work well together. They are probably three or four times as effective working together as either would be alone. And anyway, we’re only a few years into a project that could take a century or more. It’s way too early to think about risking our primary operating procedure and trusting the whole job to backups. What if the backup runs into trouble?”
“All your points are well taken,” said Governor Kresh. “So – what are the precautions I should take in talking to them?”
“Don’t lose your temper if Unit Dee is condescending to you in some way. She doesn’t really think you’re real, after all. You are really nothing more than one of the game pieces, as far as she is concerned. Don’t be thrown off if she seems to know a great deal about you, and lets you know it. Don’t correct her if she gets something wrong, either. We’ve made various adjustments to her information files for one reason or another – some deliberate errors to make it seem like a simulation, and others we set up for some procedural reason or another. Try to remember you’re not real. That’s the main thing. As for the rest of it, you’ll be talking to her via audio on a headset, and I’ll be monitoring. If there’s anything else you need to know, I’ll cut in.”
Governor Kresh nodded thoughtfully. “Have you ever noticed, Dr. Soggdon, just how much of our energy goes into dealing with the Three Laws? Getting around them, trying to make the world conform to them?”
At first, the offhand remark shocked Soggdon. Not because she disagreed with his words – far from it – but because Kresh was willing speak them. Well, if the governor was in a mood to dabble with heresy, why not indulge in it herself? “I’ve thought that for a long time, Governor,” she said. “I think the case could be made that this world is in as much trouble as it is because of the Three Laws. They’ve made us too cautious, made us worry too much about making sure today is like yesterday, and far too timid to dare plan for tomorrow.”
Kresh laughed. “Not a bad line, that,” he said. “You might catch me stealing it for use in a speech one of these fine days. “The governor looked from the Unit Dee Controller to the Unit Dum Controller, and then back up at Soggdon. “All right,” he said. “Let’s get this thing set up.”
“Goood morrn-ing. Govvvenorr Kressh.” Two voices came through the headphone to address him in unison – one a light, feminine soprano, the other a gravelly, slightly slurred, and genderless alto. They spoke the same words at the same time, but they did not synchronize with each other exactly.
The voices seemed to be coming from out of nowhere at all. No doubt that was an audio illusion produced by the stereo effect of the headphones, but it was nonetheless disconcerting. Alvar Kresh frowned and looked behind himself, as if he expected there to be two robots there, one standing behind each ear. He knew perfectly well there would be nothing to see, but there was some part of him that had to check all the same.
The whole setup seemed lunatic, irrational – but the iron hand of the Three Laws dictated that there be some such arrangement. Kresh decided to make the best of it. “Good morning,” he said, speaking into the headset’s microphone. “I take it I am addressing both Unit Dee and Unit Dum?”
“Thaat izz correct, Governorrr,” the two voices replied. “Somme vizzzitors finnnd iiit dissconcerrrting to hear usss both. Shalll we filllter ouut onne voice?”
“That might be helpful,” Kresh said. Disconcerting was far too mild a word. The two voices speaking as one was downright eerie.
“Very well,” the feminine voice said in his left ear, by itself, speaking with a sudden brisk, clipped tone, a jarring change from what had come before. Perhaps she found it easier to speak without the need to synchronize with Unit Dum. “Both of us are still on-line to you, but you will hear only one of us at a time. We will shift from one speaker to the other from time to time to remind you of our dual presence.” The voice he heard was almost excessively cheerful, with an oddly youthful tone to it. A playful voice, full of amusement and good humor.
“This higher-pitched voice I hear now,” Kresh said, “it is Unit Dee?”
“That is correct, sir.”
Suddenly the other voice, low-pitched, impersonal and slightly slurred, spoke into his right ear. “This is the voice of Unit Dum.”
“Good. Fine. Whatever. I need to speak with you both.”
“Please go ahead, Governor,” said Unit Dee in his left ear again. Kresh began to wonder if the voice-switching was some sort of game Unit Dee was playing, a way of putting him off his stride. If so, it was not going to work.
“I intend to,” he said. “I want to talk to you about an old project, from the period of the first effort to terraform this world!’
“And what would that be?” asked Unit Dee.
“The proposal to create a Polar Sea as a means of moderating planetary temperatures. I want you to consider an idea based on that old concept.”
“Ready to accept input,” said the gravelly, mechanical voice in his right ear. It was plain that very little effort had gone into giving Unit Dum a simulated personality. That was, perhaps, just as well. Kresh had the sense of talking to a schizophrenic as it was.
“Here is the idea. Assume that, in the present day, the existing Polar Depression were flooded, with inlets to the Southern Ocean provided by cutting a canal through the Utopia region on the eastern side of Terra Grande, and by redirecting the flow of the River Lethe in the west. Assume the work could be done very rapidly, within a few years’ time.”
There was the briefest of pauses. “This would cause a Polar Sea to form,” Unit Dum went on. “However, the concept is implausible. There is no way of performing such an enormous engineering task in any practical length of time.”
“Even if we could do it, I’m sure the collateral damage to existing ecosystems and property would be huge,” said Unit Dee, clearly talking more to Unit Dum than to Kresh.
“Current projections show the issues of damage to ecosystems and property become moot in between two and two point five standard centuries,” Unit Dum replied.
“Why do they become moot?” Kresh asked, fearing the answer.
“Because,” Unit Dee replied, her voice clearly unhappy, “our current projection shows all ecosystems collapsing and all humans – the owners of the property – either dying or being evacuated from the planet by that time.”
Kresh was genuinely surprised. “I was not aware that the numbers were that bad. I thought we at least had a ch
ance at survival.”
“Oh, yes,” said Unit Dee. “There is at least a chance human life will survive here. That is in large degree a matter of choice for your descendants. Human beings can survive on a lifeless, airless, sterile ball of rock if they choose to do so. If the city of Hades were domed over or rebuilt underground, and properly shielded, it could no doubt sustain a reduced population indefinitely after the climate collapses.”
“But things are improving” ‘Kresh protested. “We’re turning things around!”
“So you are – for the moment, in localized areas. But there is little or no doubt that the current short-term improvements cannot be sustained and extended in the longer term. There is simply not enough labor or equipment to expand the zones of improved climate far enough, and establish them firmly enough, for them to be self-sustaining.”
“And therefore there is no real point in worrying over ecological damage or property loss,” Kresh said. “Fine. Disregard those two points – or, rather, factor in the results of attempting to deal with them, of efforts to repair the damage.”
“The calculation involves a near-infinite number of variables,” said Unit Dum. “Recommend a pre screening process to select range of near best-case scenarios and eliminate obviously failed variants.”
“Approved,” Kresh said.
“Even the prescreening process will take a few minutes,” said Unit Dee. “Please stand by.”
“As if I had much choice,” Kresh said to no one in particular. He sat there, looking from Unit Dee’s smooth and perfect hemispherical enclosure to the boxy, awkward, hard-edged looking enclosure around Unit Dum. Dum’s enclosure, or containment, or whatever, at least had the merit of looking like machinery. Dum looked like it did something, was hooked into things, made things happen. It was hardware and wires. It was solid, firmly attached to reality by power cables and datastreams. Dum was of this world.