Rationality- From AI to Zombies

Home > Science > Rationality- From AI to Zombies > Page 18
Rationality- From AI to Zombies Page 18

by Eliezer Yudkowsky


  One afternoon, a man richly attired in white robes, leafy laurels, sandals, and business suit trudges in along the sandy trail that leads to my pastures.

  “Can I help you?” I inquire.

  The man takes a badge from his coat and flips it open, proving beyond the shadow of a doubt that he is Markos Sophisticus Maximus, a delegate from the Senate of Rum. (One might wonder whether another could steal the badge; but so great is the power of these badges that if any other were to use them, they would in that instant be transformed into Markos.)

  “Call me Mark,” he says. “I’m here to confiscate the magic pebbles, in the name of the Senate; artifacts of such great power must not fall into ignorant hands.”

  “That bleedin’ apprentice,” I grouse under my breath, “he’s been yakkin’ to the villagers again.” Then I look at Mark’s stern face, and sigh. “They aren’t magic pebbles,” I say aloud. “Just ordinary stones I picked up from the ground.”

  A flicker of confusion crosses Mark’s face, then he brightens again. “I’m here for the magic bucket!” he declares.

  “It’s not a magic bucket,” I say wearily. “I used to keep dirty socks in it.”

  Mark’s face is puzzled. “Then where is the magic?” he demands.

  An interesting question. “It’s hard to explain,” I say.

  My current apprentice, Autrey, attracted by the commotion, wanders over and volunteers his explanation: “It’s the level of pebbles in the bucket,” Autrey says. “There’s a magic level of pebbles, and you have to get the level just right, or it doesn’t work. If you throw in more pebbles, or take some out, the bucket won’t be at the magic level anymore. Right now, the magic level is,” Autrey peers into the bucket, “about one-third full.”

  “I see!” Mark says excitedly. From his back pocket Mark takes out his own bucket, and a heap of pebbles. Then he grabs a few handfuls of pebbles, and stuffs them into the bucket. Then Mark looks into the bucket, noting how many pebbles are there. “There we go,” Mark says, “the magic level of this bucket is half full. Like that?”

  “No!” Autrey says sharply. “Half full is not the magic level. The magic level is about one-third. Half full is definitely unmagic. Furthermore, you’re using the wrong bucket.”

  Mark turns to me, puzzled. “I thought you said the bucket wasn’t magic?”

  “It’s not,” I say. A sheep passes out through the gate, and I toss another pebble into the bucket. “Besides, I’m watching the sheep. Talk to Autrey.”

  Mark dubiously eyes the pebble I tossed in, but decides to temporarily shelve the question. Mark turns to Autrey and draws himself up haughtily. “It’s a free country,” Mark says, “under the benevolent dictatorship of the Senate, of course. I can drop whichever pebbles I like into whatever bucket I like.”

  Autrey considers this. “No you can’t,” he says finally, “there won’t be any magic.”

  “Look,” says Mark patiently, “I watched you carefully. You looked in your bucket, checked the level of pebbles, and called that the magic level. I did exactly the same thing.”

  “That’s not how it works,” says Autrey.

  “Oh, I see,” says Mark, “It’s not the level of pebbles in my bucket that’s magic, it’s the level of pebbles in your bucket. Is that what you claim? What makes your bucket so much better than mine, huh?”

  “Well,” says Autrey, “if we were to empty your bucket, and then pour all the pebbles from my bucket into your bucket, then your bucket would have the magic level. There’s also a procedure we can use to check if your bucket has the magic level, if we know that my bucket has the magic level; we call that a bucket compare operation.”

  Another sheep passes, and I toss in another pebble.

  “He just tossed in another pebble!” Mark says. “And I suppose you claim the new level is also magic? I could toss pebbles into your bucket until the level was the same as mine, and then our buckets would agree. You’re just comparing my bucket to your bucket to determine whether you think the level is ‘magic’ or not. Well, I think your bucket isn’t magic, because it doesn’t have the same level of pebbles as mine. So there!”

  “Wait,” says Autrey, “you don’t understand—”

  “By ‘magic level,’ you mean simply the level of pebbles in your own bucket. And when I say ‘magic level,’ I mean the level of pebbles in my bucket. Thus you look at my bucket and say it ‘isn’t magic,’ but the word ‘magic’ means different things to different people. You need to specify whose magic it is. You should say that my bucket doesn’t have ‘Autrey’s magic level,’ and I say that your bucket doesn’t have ‘Mark’s magic level.’ That way, the apparent contradiction goes away.”

  “But—” says Autrey helplessly.

  “Different people can have different buckets with different levels of pebbles, which proves this business about ‘magic’ is completely arbitrary and subjective.”

  “Mark,” I say, “did anyone tell you what these pebbles do?”

  “Do?” says Mark. “I thought they were just magic.”

  “If the pebbles didn’t do anything,” says Autrey, “our ISO 9000 process efficiency auditor would eliminate the procedure from our daily work.”

  “What’s your auditor’s name?”

  “Darwin,” says Autrey.

  “Hm,” says Mark. “Charles does have a reputation as a strict auditor. So do the pebbles bless the flocks, and cause the increase of sheep?”

  “No,” I say. “The virtue of the pebbles is this; if we look into the bucket and see the bucket is empty of pebbles, we know the pastures are likewise empty of sheep. If we do not use the bucket, we must search and search until dark, lest one last sheep remain. Or if we stop our work early, then sometimes the next morning we find a dead sheep, for the wolves savage any sheep left outside. If we look in the bucket, we know when all the sheep are home, and we can retire without fear.”

  Mark considers this. “That sounds rather implausible,” he says eventually. “Did you consider using divination sticks? Divination sticks are infallible, or at least, anyone who says they are fallible is burned at the stake. This is an extremely painful way to die; it follows that divination sticks are infallible.”

  “You’re welcome to use divination sticks if you like,” I say.

  “Oh, good heavens, of course not,” says Mark. “They work infallibly, with absolute perfection on every occasion, as befits such blessed instruments; but what if there were a dead sheep the next morning? I only use the divination sticks when there is no possibility of their being proven wrong. Otherwise I might be burned alive. So how does your magic bucket work?”

  How does the bucket work . . . ? I’d better start with the simplest possible case. “Well,” I say, “suppose the pastures are empty, and the bucket isn’t empty. Then we’ll waste hours looking for a sheep that isn’t there. And if there are sheep in the pastures, but the bucket is empty, then Autrey and I will turn in too early, and we’ll find dead sheep the next morning. So an empty bucket is magical if and only if the pastures are empty—”

  “Hold on,” says Autrey. “That sounds like a vacuous tautology to me. Aren’t an empty bucket and empty pastures obviously the same thing?”

  “It’s not vacuous,” I say. “Here’s an analogy: The logician Alfred Tarski once said that the assertion ‘Snow is white’ is true if and only if snow is white. If you can understand that, you should be able to see why an empty bucket is magical if and only if the pastures are empty of sheep.”

  “Hold on,” says Mark. “These are buckets. They don’t have anything to do with sheep. Buckets and sheep are obviously completely different. There’s no way the sheep can ever interact with the bucket.”

  “Then where do you think the magic comes from?” inquires Autrey.

  Mark considers. “You said you could compare two buckets to check if they had the same level . . . I can see how buckets can interact with buckets. Maybe when you get a large collection of buckets, and they all have the
same level, that’s what generates the magic. I’ll call that the coherentist theory of magic buckets.”

  “Interesting,” says Autrey. “I know that my master is working on a system with multiple buckets—he says it might work better because of ‘redundancy’ and ‘error correction.’ That sounds like coherentism to me.”

  “They’re not quite the same—” I start to say.

  “Let’s test the coherentism theory of magic,” says Autrey. “I can see you’ve got five more buckets in your back pocket. I’ll hand you the bucket we’re using, and then you can fill up your other buckets to the same level—”

  Mark recoils in horror. “Stop! These buckets have been passed down in my family for generations, and they’ve always had the same level! If I accept your bucket, my bucket collection will become less coherent, and the magic will go away!”

  “But your current buckets don’t have anything to do with the sheep!” protests Autrey.

  Mark looks exasperated. “Look, I’ve explained before, there’s obviously no way that sheep can interact with buckets. Buckets can only interact with other buckets.”

  “I toss in a pebble whenever a sheep passes,” I point out.

  “When a sheep passes, you toss in a pebble?” Mark says. “What does that have to do with anything?”

  “It’s an interaction between the sheep and the pebbles,” I reply.

  “No, it’s an interaction between the pebbles and you,” Mark says. “The magic doesn’t come from the sheep, it comes from you. Mere sheep are obviously nonmagical. The magic has to come from somewhere, on the way to the bucket.”

  I point at a wooden mechanism perched on the gate. “Do you see that flap of cloth hanging down from that wooden contraption? We’re still fiddling with that—it doesn’t work reliably—but when sheep pass through, they disturb the cloth. When the cloth moves aside, a pebble drops out of a reservoir and falls into the bucket. That way, Autrey and I won’t have to toss in the pebbles ourselves.”

  Mark furrows his brow. “I don’t quite follow you . . . is the cloth magical?”

  I shrug. “I ordered it online from a company called Natural Selections. The fabric is called Sensory Modality.” I pause, seeing the incredulous expressions of Mark and Autrey. “I admit the names are a bit New Agey. The point is that a passing sheep triggers a chain of cause and effect that ends with a pebble in the bucket. Afterward you can compare the bucket to other buckets, and so on.”

  “I still don’t get it,” Mark says. “You can’t fit a sheep into a bucket. Only pebbles go in buckets, and it’s obvious that pebbles only interact with other pebbles.”

  “The sheep interact with things that interact with pebbles . . .” I search for an analogy. “Suppose you look down at your shoelaces. A photon leaves the Sun; then travels down through Earth’s atmosphere; then bounces off your shoelaces; then passes through the pupil of your eye; then strikes the retina; then is absorbed by a rod or a cone. The photon’s energy makes the attached neuron fire, which causes other neurons to fire. A neural activation pattern in your visual cortex can interact with your beliefs about your shoelaces, since beliefs about shoelaces also exist in neural substrate. If you can understand that, you should be able to see how a passing sheep causes a pebble to enter the bucket.”

  “At exactly which point in the process does the pebble become magic?” says Mark.

  “It . . . um . . .” Now I’m starting to get confused. I shake my head to clear away cobwebs. This all seemed simple enough when I woke up this morning, and the pebble-and-bucket system hasn’t gotten any more complicated since then. “This is a lot easier to understand if you remember that the point of the system is to keep track of sheep.”

  Mark sighs sadly. “Never mind . . . it’s obvious you don’t know. Maybe all pebbles are magical to start with, even before they enter the bucket. We could call that position panpebblism.”

  “Ha!” Autrey says, scorn rich in his voice. “Mere wishful thinking! Not all pebbles are created equal. The pebbles in your bucket are not magical. They’re only lumps of stone!”

  Mark’s face turns stern. “Now,” he cries, “now you see the danger of the road you walk! Once you say that some people’s pebbles are magical and some are not, your pride will consume you! You will think yourself superior to all others, and so fall! Many throughout history have tortured and murdered because they thought their own pebbles supreme!” A tinge of condescension enters Mark’s voice. “Worshipping a level of pebbles as ‘magical’ implies that there’s an absolute pebble level in a Supreme Bucket. Nobody believes in a Supreme Bucket these days.”

  “One,” I say. “Sheep are not absolute pebbles. Two, I don’t think my bucket actually contains the sheep. Three, I don’t worship my bucket level as perfect—I adjust it sometimes—and I do that because I care about the sheep.”

  “Besides,” says Autrey, “someone who believes that possessing absolute pebbles would license torture and murder, is making a mistake that has nothing to do with buckets. You’re solving the wrong problem.”

  Mark calms himself down. “I suppose I can’t expect any better from mere shepherds. You probably believe that snow is white, don’t you.”

  “Um . . . yes?” says Autrey.

  “It doesn’t bother you that Joseph Stalin believed that snow is white?”

  “Um . . . no?” says Autrey.

  Mark gazes incredulously at Autrey, and finally shrugs. “Let’s suppose, purely for the sake of argument, that your pebbles are magical and mine aren’t. Can you tell me what the difference is?”

  “My pebbles represent the sheep!” Autrey says triumphantly. “Your pebbles don’t have the representativeness property, so they won’t work. They are empty of meaning. Just look at them. There’s no aura of semantic content; they are merely pebbles. You need a bucket with special causal powers.”

  “Ah!” Mark says. “Special causal powers, instead of magic.”

  “Exactly,” says Autrey. “I’m not superstitious. Postulating magic, in this day and age, would be unacceptable to the international shepherding community. We have found that postulating magic simply doesn’t work as an explanation for shepherding phenomena. So when I see something I don’t understand, and I want to explain it using a model with no internal detail that makes no predictions even in retrospect, I postulate special causal powers. If that doesn’t work, I’ll move on to calling it an emergent phenomenon.”

  “What kind of special powers does the bucket have?” asks Mark.

  “Hm,” says Autrey. “Maybe this bucket is imbued with an about-ness relation to the pastures. That would explain why it worked—when the bucket is empty, it means the pastures are empty.”

  “Where did you find this bucket?” says Mark. “And how did you realize it had an about-ness relation to the pastures?”

  “It’s an ordinary bucket,” I say. “I used to climb trees with it . . . I don’t think this question needs to be difficult.”

  “I’m talking to Autrey,” says Mark.

  “You have to bind the bucket to the pastures, and the pebbles to the sheep, using a magical ritual—pardon me, an emergent process with special causal powers—that my master discovered,” Autrey explains.

  Autrey then attempts to describe the ritual, with Mark nodding along in sage comprehension.

  “You have to throw in a pebble every time a sheep leaves through the gate?” says Mark. “Take out a pebble every time a sheep returns?”

  Autrey nods. “Yeah.”

  “That must be really hard,” Mark says sympathetically.

  Autrey brightens, soaking up Mark’s sympathy like rain. “Exactly!” says Autrey. “It’s extremely hard on your emotions. When the bucket has held its level for a while, you . . . tend to get attached to that level.”

  A sheep passes then, leaving through the gate. Autrey sees; he stoops, picks up a pebble, holds it aloft in the air. “Behold!” Autrey proclaims. “A sheep has passed! I must now toss a pebble into this bucket, my d
ear bucket, and destroy that fond level which has held for so long—” Another sheep passes. Autrey, caught up in his drama, misses it; so I plunk a pebble into the bucket. Autrey is still speaking: “—for that is the supreme test of the shepherd, to throw in the pebble, be it ever so agonizing, be the old level ever so precious. Indeed, only the best of shepherds can meet a requirement so stern—”

  “Autrey,” I say, “if you want to be a great shepherd someday, learn to shut up and throw in the pebble. No fuss. No drama. Just do it.”

  “And this ritual,” says Mark, “it binds the pebbles to the sheep by the magical laws of Sympathy and Contagion, like a voodoo doll.”

  Autrey winces and looks around. “Please! Don’t call it Sympathy and Contagion. We shepherds are an anti-superstitious folk. Use the word ‘intentionality,’ or something like that.”

  “Can I look at a pebble?” says Mark.

  “Sure,” I say. I take one of the pebbles out of the bucket, and toss it to Mark. Then I reach to the ground, pick up another pebble, and drop it into the bucket.

  Autrey looks at me, puzzled. “Didn’t you just mess it up?”

  I shrug. “I don’t think so. We’ll know I messed it up if there’s a dead sheep next morning, or if we search for a few hours and don’t find any sheep.”

  “But—” Autrey says.

  “I taught you everything you know, but I haven’t taught you everything I know,” I say.

  Mark is examining the pebble, staring at it intently. He holds his hand over the pebble and mutters a few words, then shakes his head. “I don’t sense any magical power,” he says. “Pardon me. I don’t sense any intentionality.”

 

‹ Prev