by Isaac Asimov
"You told him to go away?" asked Dr. Calvin with sharp interest. "In just those words? Did you say `Go away'? Try to remember the exact words."
There was apparently an internal struggle in progress. Black cradled his forehead in a broad palm for a moment, then tore it away and said defiantly, "I said, 'Go lose yourself.' "
Bogert laughed for a short moment. "And he did, eh?"
But Calvin wasn't finished. She spoke cajolingly, "Now we're getting somewhere, Mr. Black. But exact details are important. In understanding the robot's actions, a word, a gesture, an emphasis may be everything. You couldn't have said just those three words, for instance, could you? By your own description you must have been in a hasty mood. Perhaps you strengthened your speech a little."
The young man reddened, "Well… I may have called it a… a few things."
"Exactly what things?"
"Oh – I wouldn't remember exactly. Besides I couldn't repeat it. You know how you get when you're excited." His embarrassed laugh was almost a giggle, "I sort of have a tendency to strong language."
"That's quite all right," she replied, with prim severity. "At the moment, I'm a psychologist. I would like to have you repeat exactly what you said as nearly as you remember, and, even more important, the exact tone of voice you used."
Black looked at his commanding officer for support, found none. His eyes grew round and appalled, "But I can't."
"You must."
"Suppose," said Bogert, with ill-hidden amusement, "you address me. You may find it easier."
The young man's scarlet face turned to Bogert. He swallowed. "I said" His voice faded out. He tried again, "I said-"
And he drew a deep breath and spewed it out hastily in one long succession of syllables. Then, in the charged air that lingered, he concluded almost in tears, "… more or less. I don't remember the exact order of what I called him, and maybe I left out something or put in something, but that was about it."
Only the slightest flush betrayed any feeling on the part of the robopsychologist. She said, "I am aware of the meaning of most of the terms used. The others, I suppose, are equally derogatory."
"I'm afraid so," agreed the tormented Black.
"And in among it, you told him to lose himself."
"I meant it only figuratively."
"I realize that. No disciplinary action is intended, I am sure." And at her glance, the general, who, five seconds earlier, had seemed not sure at all, nodded angrily.
"You may leave, Mr. Black. Thank you for your cooperation."
It took five hours for Susan Calvin to interview the sixty-three robots. It was five hours of multi-repetition; of replacement after replacement of identical robot; of Questions A, B, C, D; and Answers A, B, C, D; of a carefully bland expression, a carefully neutral tone, a carefully friendly atmosphere; and a hidden wire recorder.
The psychologist felt drained of vitality when she was finished.
Bogert was waiting for her and looked expectant as she dropped the recording spool with a clang upon the plastic of the desk.
She shook her head, "All sixty-three seemed the same to me. I couldn't tell-"
He said, "You couldn't expect to tell by ear, Susan. Suppose we analyze the recordings."
Ordinarily, the mathematical interpretation of verbal reactions of robots is one of the more intricate branches of robotic analysis. It requires a staff of trained technicians and the help of complicated computing machines. Bogert knew that. Bogert stated as much, in an extreme of unshown annoyance after having listened to each set of replies, made lists of word deviations, and graphs of the intervals of responses.
"There are no anomalies present, Susan. The variations in wording and the time reactions are within the limits of ordinary frequency groupings. We need finer methods. They must have computers here. No." He frowned and nibbled delicately at a thumbnail. "We can't use computers. Too much danger of leakage. Or maybe if we-"
Dr. Calvin stopped him with an impatient gesture, "Please, Peter. This isn't one of your petty laboratory problems. If we can't determine the modified Nestor by some gross difference that we can see with the naked eye, one that there is no mistake about, we're out of luck. The danger of being wrong, and of letting him escape is otherwise too great. It's not enough to point out a minute irregularity in a graph. I tell you, if that's all I've got to go on, I'd destroy them all just to be certain. Have you spoken to the other modified Nestors?"
"Yes, I have," snapped back Bogert, "and there's nothing wrong with them. They're above normal in friendliness if anything. They answered my questions, displayed pride in their knowledge – except the two new ones that haven't had time to learn their etheric physics. They laughed rather good-naturedly at my ignorance in some of the specializations here." He shrugged, "I suppose that forms some of the basis for resentment toward them on the part of the technicians here. The robots are perhaps too willing to impress you with their greater knowledge."
"Can you try a few Planar Reactions to see if there has been any change, any deterioration, in their mental set-up since manufacture?"
"I haven't yet, but I will." He shook a slim finger at her, "You're losing your nerve, Susan. I don't see what it is you're dramatizing. They're essentially harmless."
"They are?" Calvin took fire. "They are? Do you realize one of them is lying? One of the sixty-three robots I have just interviewed has deliberately lied to me after the strictest injunction to tell the truth. The abnormality indicated is horribly deep-seated, and horribly frightening."
Peter Bogert felt his teeth harden against each other. He said, "Not at all. Look! Nestor 10 was given orders to lose himself. Those orders were expressed in maximum urgency by the person most authorized to command him. You can't counteract that order either by superior urgency or superior right of command. Naturally, the robot will attempt to defend the carrying out of his orders. In fact, objectively, I admire his ingenuity. How better can a robot lose himself than to hide himself among a group of similar robots?"
"Yes, you would admire it. I've detected amusement in you, Peter – amusement and an appalling lack of understanding. Are you a roboticist, Peter? Those robots attach importance to what they consider superiority. You've just said as much yourself. Subconsciously they feel humans to be inferior and the First Law which protects us from them is imperfect. They are unstable. And here we have a young man ordering a robot to leave him, to lose himself, with every verbal appearance of revulsion, disdain, and disgust. Granted, that robot must follow orders, but subconsciously, there is resentment. It will become more important than ever for it to prove that it is superior despite the horrible names it was called. It may become so important that what's left of the First Law won't be enough."
"How on Earth, or anywhere in the Solar System, Susan, is a robot going to know the meaning of the assorted strong language used upon him? Obscenity is not one of the things impressioned upon his brain."
"Original impressionment is not everything," Calvin snarled at him. "Robots have learning capacity, you… you fool-" And Bogert knew that she had really lost her temper. She continued hastily, "Don't you suppose he could tell from the tone used that the words weren't complimentary? Don't yon suppose he's heard the words used before and noted upon what occasions?"
"Well, then," shouted Bogert, "will you kindly tell me one way in which a modified robot can harm a human being, no matter how offended it is, no matter how sick with desire to prove superiority?"
"If I tell you one way, will you keep quiet?"
"Yes."
They were leaning across the table at each other, angry eyes nailed together.
The psychologist said, "If a modified robot were to drop a heavy weight upon a human being, he would not be breaking the First Law, if he did so with the knowledge that his strength and reaction speed would be sufficient to snatch the weight away before it struck the man. However once the weight left his fingers, he would be no longer the active medium. Only the blind force of gravity would be
that. The robot could then change his mind and merely by inaction, allow the weight to strike. The modified First Law allows that."
"That's an awful stretch of imagination."
"That's what my profession requires sometimes. Peter, let's not quarrel. Let's work. You know the exact nature of the stimulus that caused the robot to lose himself. You have the records of his original mental make-up. I want you to tell me how possible it is for our robot to do the sort of thing I just talked about. Not the specific instance, mind you, but that whole class of response. And I want it done quickly."
"And meanwhile-"
"And meanwhile, we'll have to try performance tests directly on the response to First Law."
Gerald Black, at his own request, was supervising the mushrooming wooden partitions that were springing up in a bellying circle on the vaulted third floor of Radiation Building 2. The laborers worked, in the main, silently, but more than one was openly a-wonder at the sixty-three photocells that required installation.
One of them sat down near Black, removed his hat, and wiped his forehead thoughtfully with a freckled forearm.
Black nodded at him, "How's it doing, Walensky?"
Walensky shrugged and fired a cigar, "Smooth as butter. What's going on anyway, Doc? First, there's no work for three days and then we have this mess of jiggers." He leaned backward on his elbows and puffed smoke.
Black twitched his eyebrows, "A couple of robot men came over from Earth. Remember the trouble we had with robots running into the gamma fields. before we pounded it into their skulls that they weren't to do it."
"Yeah. Didn't we get new robots?"
"We got some replacements, but mostly it was a job of indoctrination. Anyway, the people who make them want to figure out robots that aren't hit so bad by gamma rays."
"Sure seems funny, though, to stop all the work on the Drive for this robot deal. I thought nothing was allowed to stop the Drive."
"Well, it's the fellows upstairs that have the say on that. Me- I just do as I'm told. Probably all a matter of pull-"
"Yeah," the electrician jerked a smile, and winked a wise eye. "Somebody knew somebody in Washington. But as long as my pay comes through on the dot, I should worry. The Drive's none of my affair. What are they going to do here?"
"You're asking me? They brought a mess of robots with them, -over sixty, and they're going to measure reactions. That's all my knowledge."
"How long will it take?"
"I wish I knew."
"Well," Walensky said, with heavy sarcasm, "as long as they dish me my money, they can play games all they want."
Black felt quietly satisfied. Let the story spread. It was harmless, and near enough to the truth to take the fangs out of curiosity.
A man sat in the chair, motionless, silent. A weight dropped, crashed downward, then pounded aside at the last moment under the synchronized thump of a sudden force beam. In sixty-three wooden cells, watching NS-2 robots dashed forward in that split second before the weight veered, and sixty-three photocells five feet ahead of their original positions jiggled the marking pen and presented a little jag on the paper. The weight rose and dropped, rose and dropped, rose-
Ten times!
Ten times the robots sprang forward and stopped, as the man remained safely seated.
Major-general Kallner had not worn his uniform in its entirety since the first dinner with the U. S. Robot representatives. He wore nothing over his blue-gray shirt now, the collar was open, and the black tie was pulled loose.
He looked hopefully at Bogert, who was still blandly neat and whose inner tension was perhaps betrayed only by the trace of glister at his temples.
The general said, "How does it look? What is it you're trying to see?"
Bogert replied, "A difference which may turn out to be a little too subtle for our purposes, I'm afraid. For sixty-two of those robots the necessity of jumping toward the apparently threatened human was what we call, in robotics, a forced reaction. You see, even when the robots knew that the human in question would not come to harm – and after the third or fourth time they must have known it – they could not prevent reacting as they did. First Law requires it"
"Well?"
"But the sixty-third robot, the modified Nestor, had no such complusion. He was under free action. If he had wished, he could have remained in his seat. Unfortunately," said his voice was mildly regretful, "he didn't so wish."
"Why do you suppose?"
Bogert shrugged, "I suppose Dr. Calvin will tell us when she gets here. Probably with a horribly pessimistic interpretation, too. She is sometimes a bit annoying."
"She's qualified, isn't she?" demanded the general with a sudden frown of uneasiness.
"Yes." Bogert seemed amused. "She's qualified all right. She understands robots like a sister – comes from hating human beings so much, I think. It's just that, psychologist or not, she's an extreme neurotic. Has paranoid tendencies. Don't take her too seriously."
He spread the long row of broken-line graphs out in front of him. "You see, general, in the case of each robot the time interval from moment of drop to the completion of a five-foot movement tends to decrease as the tests are repeated. There's a definite mathematical relationship that governs such things and failure to conform would indicate marked abnormality in the positronic brain. Unfortunately, all here appear normal."
"But if our Nestor 10 was not responding with a forced action, why isn't his curve different? I don't understand that."
"It's simple enough. Robotic responses are not perfectly analogous to human responses, more's the pity. In human beings, voluntary action is much slower than reflex action. But that's not the case with robots; with them it is merely a question of freedom of choice, otherwise the speeds of free and forced action are much the same. What I had been expecting, though, was that Nestor 10 would be caught by surprise the first time and allow too great an interval to elapse before responding."
"And he didn't?"
"I'm afraid not."
"Then we haven't gotten anywhere." The general sat back with an expression of pain. "It's five days since you've come."
At this point, Susan Calvin entered and slammed the door behind her. "Put your graphs away, Peter," she cried, "you know they don't show anything."
She mumbled something impatiently as Kallner half-rose to greet her, and went on, "We'll have to try something else quickly. I don't like what's happening."
Bogert exchanged a resigned glance with the general. "Is anything wrong?"
"You mean specifically? No. But I don't like to have Nestor 10 continue to elude us. It's bad. It must be gratifying his swollen sense of superiority. I'm afraid that his motivation is no longer simply one of following orders. I think it's becoming more a matter of sheer neurotic necessity to outthink humans. That's a dangerously unhealthy situation. Peter, have you done what I asked? Have you worked out the instability factors of the modified NS-2 along the lines I want?"
"It's in progress," said the mathematician, without interest.
She stared at him angrily for a moment, then turned to Kallner. "Nester 10 is decidedly aware of what we're doing, general. He had no reason to jump for the bait in this experiment, especially after the first time, when he must have seen that there was no real danger to our subject. The others couldn't help it; but he was deliberately falsifying a reaction."
"What do you think we ought to do now, then, Dr. Calvin?"
"Make it impossible for him to fake an action the next time. We will repeat the experiment, but with an addition. High-tension cables, capable of electrocuting the Nestor models will be placed between subject and robot – enough of them to avoid the possibility of jumping over – and the robot will be made perfectly aware in advance that touching the cables will mean death."
"Hold on," spat out Bogert with sudden viciousness. "I rule that out. We are not electrocuting two million dollars worth of robots to locate Nestor 10. There are other ways."
"You're certain? You've
found none. In any case, it's not a question of electrocution. We can arrange a relay which will break the current at the instant of application of weight. If the robot should place his weight on it, he won't die. But he won't know that, you see."
The general's eyes gleamed into hope. "Will that work?"
"It should. Under those conditions, Nestor 10 would have to remain in his seat. He could be ordered to touch the cables and die, for the Second Law of obedience is superior to the Third Law of self-preservation. But he won't be ordered to; he will merely be left to his own devices, as will all the robots. In the case of the normal robots, the First Law of human safety will drive them to their death even without orders. But not our Nestor 10. Without the entire First Law, and without having received any orders on the matter, the Third Law, self-preservation, will be the highest operating, and he will have no choice but to remain in his seat. It would be a forced action."
"Will it be done tonight, then?"
"Tonight," said the psychologist, "if the cables can be laid In time. I'll tell the robots now what they're to be up against…
A man sat in the chair, motionless, silent. A weight dropped, crashed downward, then pounded aside at the last moment under the synchronized thump of a sudden force beam.
Only once-
And from her small camp chair in the observing booth in the balcony, Dr. Susan Calvin rose with a short gasp of pure horror.
Sixty-three robots sat quietly in their chairs, staring owlishly at the endangered man before them. Not one moved.
Dr. Calvin was angry, angry almost past endurance. Angry the worse for not daring to show it to the robots that, one by one, were entering the room and then leaving. She checked the list. Number twenty-eight was due in now – Thirty-five still lay ahead of her.
Number Twenty-eight entered, diffidently.
She forced herself into reasonable calm. "And who are You?"
The robot replied in a low, uncertain voice, "I have received no number of my own yet, ma'am. I'm an NS-2 robot, and I was Number Twenty-eight in line outside. I have a slip of paper here that I'm to give to you."