Rationality- From AI to Zombies

Home > Science > Rationality- From AI to Zombies > Page 19
Rationality- From AI to Zombies Page 19

by Eliezer Yudkowsky


  “A pebble only has intentionality if it’s inside a ma—an emergent bucket,” says Autrey. “Otherwise it’s just a mere pebble.”

  “Not a problem,” I say. I take a pebble out of the bucket, and toss it away. Then I walk over to where Mark stands, tap his hand holding a pebble, and say: “I declare this hand to be part of the magic bucket!” Then I resume my post at the gates.

  Autrey laughs. “Now you’re just being gratuitously evil.”

  I nod, for this is indeed the case.

  “Is that really going to work, though?” says Autrey.

  I nod again, hoping that I’m right. I’ve done this before with two buckets, and in principle, there should be no difference between Mark’s hand and a bucket. Even if Mark’s hand is imbued with the élan vital that distinguishes live matter from dead matter, the trick should work as well as if Mark were a marble statue.

  Mark is looking at his hand, a bit unnerved. “So . . . the pebble has intentionality again, now?”

  “Yep,” I say. “Don’t add any more pebbles to your hand, or throw away the one you have, or you’ll break the ritual.”

  Mark nods solemnly. Then he resumes inspecting the pebble. “I understand now how your flocks grew so great,” Mark says. “With the power of this bucket, you could keep on tossing pebbles, and the sheep would keep returning from the fields. You could start with just a few sheep, let them leave, then fill the bucket to the brim before they returned. And if tending so many sheep grew tedious, you could let them all leave, then empty almost all the pebbles from the bucket, so that only a few returned . . . increasing the flocks again when it came time for shearing . . . dear heavens, man! Do you realize the sheer power of this ritual you’ve discovered? I can only imagine the implications; humankind might leap ahead a decade—no, a century!”

  “It doesn’t work that way,” I say. “If you add a pebble when a sheep hasn’t left, or remove a pebble when a sheep hasn’t come in, that breaks the ritual. The power does not linger in the pebbles, but vanishes all at once, like a soap bubble popping.”

  Mark’s face is terribly disappointed. “Are you sure?”

  I nod. “I tried that and it didn’t work.”

  Mark sighs heavily. “And this . . . math . . . seemed so powerful and useful until then . . . Oh, well. So much for human progress.”

  “Mark, it was a brilliant idea,” Autrey says encouragingly. “The notion didn’t occur to me, and yet it’s so obvious . . . it would save an enormous amount of effort . . . there must be a way to salvage your plan! We could try different buckets, looking for one that would keep the magical pow—the intentionality in the pebbles, even without the ritual. Or try other pebbles. Maybe our pebbles just have the wrong properties to have inherent intentionality. What if we tried it using stones carved to resemble tiny sheep? Or just write ‘sheep’ on the pebbles; that might be enough.”

  “Not going to work,” I predict dryly.

  Autrey continues. “Maybe we need organic pebbles, instead of silicon pebbles . . . or maybe we need to use expensive gemstones. The price of gemstones doubles every eighteen months, so you could buy a handful of cheap gemstones now, and wait, and in twenty years they’d be really expensive.”

  “You tried adding pebbles to create more sheep, and it didn’t work?” Mark asks me. “What exactly did you do?”

  “I took a handful of dollar bills. Then I hid the dollar bills under a fold of my blanket, one by one; each time I hid another bill, I took another paperclip from a box, making a small heap. I was careful not to keep track in my head, so that all I knew was that there were ‘many’ dollar bills, and ‘many’ paperclips. Then when all the bills were hidden under my blanket, I added a single additional paperclip to the heap, the equivalent of tossing an extra pebble into the bucket. Then I started taking dollar bills from under the fold, and putting the paperclips back into the box. When I finished, a single paperclip was left over.”

  “What does that result mean?” asks Autrey.

  “It means the trick didn’t work. Once I broke ritual by that single misstep, the power did not linger, but vanished instantly; the heap of paperclips and the pile of dollar bills no longer went empty at the same time.”

  “You actually tried this?” asks Mark.

  “Yes,” I say, “I actually performed the experiment, to verify that the outcome matched my theoretical prediction. I have a sentimental fondness for the scientific method, even when it seems absurd. Besides, what if I’d been wrong?”

  “If it had worked,” says Mark, “you would have been guilty of counterfeiting! Imagine if everyone did that; the economy would collapse! Everyone would have billions of dollars of currency, yet there would be nothing for money to buy!”

  “Not at all,” I reply. “By that same logic whereby adding another paperclip to the heap creates another dollar bill, creating another dollar bill would create an additional dollar’s worth of goods and services.”

  Mark shakes his head. “Counterfeiting is still a crime . . . You should not have tried.”

  “I was reasonably confident I would fail.”

  “Aha!” says Mark. “You expected to fail! You didn’t believe you could do it!”

  “Indeed,” I admit. “You have guessed my expectations with stunning accuracy.”

  “Well, that’s the problem,” Mark says briskly. “Magic is fueled by belief and willpower. If you don’t believe you can do it, you can’t. You need to change your belief about the experimental result; that will change the result itself.”

  “Funny,” I say nostalgically, “that’s what Autrey said when I told him about the pebble-and-bucket method. That it was too ridiculous for him to believe, so it wouldn’t work for him.”

  “How did you persuade him?” inquires Mark.

  “I told him to shut up and follow instructions,” I say, “and when the method worked, Autrey started believing in it.”

  Mark frowns, puzzled. “That makes no sense. It doesn’t resolve the essential chicken-and-egg dilemma.”

  “Sure it does. The bucket method works whether or not you believe in it.”

  “That’s absurd!” sputters Mark. “I don’t believe in magic that works whether or not you believe in it!”

  “I said that too,” chimes in Autrey. “Apparently I was wrong.”

  Mark screws up his face in concentration. “But . . . if you didn’t believe in magic that works whether or not you believe in it, then why did the bucket method work when you didn’t believe in it? Did you believe in magic that works whether or not you believe in it whether or not you believe in magic that works whether or not you believe in it?”

  “I don’t . . . think so . . .” says Autrey doubtfully.

  “Then if you didn’t believe in magic that works whether or not you . . . hold on a second, I need to work this out with paper and pencil—” Mark scribbles frantically, looks skeptically at the result, turns the piece of paper upside down, then gives up. “Never mind,” says Mark. “Magic is difficult enough for me to comprehend; metamagic is out of my depth.”

  “Mark, I don’t think you understand the art of bucketcraft,” I say. “It’s not about using pebbles to control sheep. It’s about making sheep control pebbles. In this art, it is not necessary to begin by believing the art will work. Rather, first the art works, then one comes to believe that it works.”

  “Or so you believe,” says Mark.

  “So I believe,” I reply, “because it happens to be a fact. The correspondence between reality and my beliefs comes from reality controlling my beliefs, not the other way around.”

  Another sheep passes, causing me to toss in another pebble.

  “Ah! Now we come to the root of the problem,” says Mark. “What’s this so-called ‘reality’ business? I understand what it means for a hypothesis to be elegant, or falsifiable, or compatible with the evidence. It sounds to me like calling a belief ‘true’ or ‘real’ or ‘actual’ is merely the difference between saying you believe something,
and saying you really really believe something.”

  I pause. “Well . . .” I say slowly. “Frankly, I’m not entirely sure myself where this ‘reality’ business comes from. I can’t create my own reality in the lab, so I must not understand it yet. But occasionally I believe strongly that something is going to happen, and then something else happens instead. I need a name for whatever-it-is that determines my experimental results, so I call it ‘reality’. This ‘reality’ is somehow separate from even my very best hypotheses. Even when I have a simple hypothesis, strongly supported by all the evidence I know, sometimes I’m still surprised. So I need different names for the thingies that determine my predictions and the thingy that determines my experimental results. I call the former thingies ‘belief,’ and the latter thingy ‘reality.’”

  Mark snorts. “I don’t even know why I bother listening to this obvious nonsense. Whatever you say about this so-called ‘reality,’ it is merely another belief. Even your belief that reality precedes your beliefs is a belief. It follows, as a logical inevitability, that reality does not exist; only beliefs exist.”

  “Hold on,” says Autrey, “could you repeat that last part? You lost me with that sharp swerve there in the middle.”

  “No matter what you say about reality, it’s just another belief,” explains Mark. “It follows with crushing necessity that there is no reality, only beliefs.”

  “I see,” I say. “The same way that no matter what you eat, you need to eat it with your mouth. It follows that there is no food, only mouths.”

  “Precisely,” says Mark. “Everything that you eat has to be in your mouth. How can there be food that exists outside your mouth? The thought is nonsense, proving that ‘food’ is an incoherent notion. That’s why we’re all starving to death; there’s no food.”

  Autrey looks down at his stomach. “But I’m not starving to death.”

  “Aha!” shouts Mark triumphantly. “And how did you utter that very objection? With your mouth, my friend! With your mouth! What better demonstration could you ask that there is no food?”

  “What’s this about starvation?” demands a harsh, rasping voice from directly behind us. Autrey and I stay calm, having gone through this before. Mark leaps a foot in the air, startled almost out of his wits.

  Inspector Darwin smiles tightly, pleased at achieving surprise, and makes a small tick on his clipboard.

  “Just a metaphor!” Mark says quickly. “You don’t need to take away my mouth, or anything like that—”

  “Why do you need a mouth if there is no food?” demands Darwin angrily. “Never mind. I have no time for this foolishness. I am here to inspect the sheep.”

  “Flock’s thriving, sir,” I say. “No dead sheep since January.”

  “Excellent. I award you 0.12 units of fitness. Now what is this person doing here? Is he a necessary part of the operations?”

  “As far as I can see, he would be of more use to the human species if hung off a hot-air balloon as ballast,” I say.

  “Ouch,” says Autrey mildly.

  “I do not care about the human species. Let him speak for himself.”

  Mark draws himself up haughtily. “This mere shepherd,” he says, gesturing at me, “has claimed that there is such a thing as reality. This offends me, for I know with deep and abiding certainty that there is no truth. The concept of ‘truth’ is merely a stratagem for people to impose their own beliefs on others. Every culture has a different ‘truth,’ and no culture’s ‘truth’ is superior to any other. This that I have said holds at all times in all places, and I insist that you agree.”

  “Hold on a second,” says Autrey. “If nothing is true, why should I believe you when you say that nothing is true?”

  “I didn’t say that nothing is true—” says Mark.

  “Yes, you did,” interjects Autrey, “I heard you.”

  “—I said that ‘truth’ is an excuse used by some cultures to enforce their beliefs on others. So when you say something is ‘true,’ you mean only that it would be advantageous to your own social group to have it believed.”

  “And this that you have said,” I say, “is it true?”

  “Absolutely, positively true!” says Mark emphatically. “People create their own realities.”

  “Hold on,” says Autrey, sounding puzzled again, “saying that people create their own realities is, logically, a completely separate issue from saying that there is no truth, a state of affairs I cannot even imagine coherently, perhaps because you still have not explained how exactly it is supposed to work—”

  “There you go again,” says Mark exasperatedly, “trying to apply your Western concepts of logic, rationality, reason, coherence, and self-consistency.”

  “Great,” mutters Autrey, “now I need to add a third subject heading, to keep track of this entirely separate and distinct claim—”

  “It’s not separate,” says Mark. “Look, you’re taking the wrong attitude by treating my statements as hypotheses, and carefully deriving their consequences. You need to think of them as fully general excuses, which I apply when anyone says something I don’t like. It’s not so much a model of how the universe works, as a Get Out of Jail Free card. The key is to apply the excuse selectively. When I say that there is no such thing as truth, that applies only to your claim that the magic bucket works whether or not I believe in it. It does not apply to my claim that there is no such thing as truth.”

  “Um . . . why not?” inquires Autrey.

  Mark heaves a patient sigh. “Autrey, do you think you’re the first person to think of that question? To ask us how our own beliefs can be meaningful if all beliefs are meaningless? That’s the same thing many students say when they encounter this philosophy, which, I’ll have you know, has many adherents and an extensive literature.”

  “So what’s the answer?” says Autrey.

  “We named it the ‘reflexivity problem,’” explains Mark.

  “But what’s the answer?” persists Autrey.

  Mark smiles condescendingly. “Believe me, Autrey, you’re not the first person to think of such a simple question. There’s no point in presenting it to us as a triumphant refutation.”

  “But what’s the actual answer?”

  “Now, I’d like to move on to the issue of how logic kills cute baby seals—”

  “You are wasting time,” snaps Inspector Darwin.

  “Not to mention, losing track of sheep,” I say, tossing in another pebble.

  Inspector Darwin looks at the two arguers, both apparently unwilling to give up their positions. “Listen,” Darwin says, more kindly now, “I have a simple notion for resolving your dispute. You say,” says Darwin, pointing to Mark, “that people’s beliefs alter their personal realities. And you fervently believe,” his finger swivels to point at Autrey, “that Mark’s beliefs can’t alter reality. So let Mark believe really hard that he can fly, and then step off a cliff. Mark shall see himself fly away like a bird, and Autrey shall see him plummet down and go splat, and you shall both be happy.”

  We all pause, considering this.

  “It sounds reasonable . . .” Mark says finally.

  “There’s a cliff right there,” observes Inspector Darwin.

  Autrey is wearing a look of intense concentration. Finally he shouts: “Wait! If that were true, we would all have long since departed into our own private universes, in which case the other people here are only figments of your imagination—there’s no point in trying to prove anything to us—”

  A long dwindling scream comes from the nearby cliff, followed by a dull and lonely splat. Inspector Darwin flips his clipboard to the page that shows the current gene pool and pencils in a slightly lower frequency for Mark’s alleles.

  Autrey looks slightly sick. “Was that really necessary?”

  “Necessary?” says Inspector Darwin, sounding puzzled. “It just happened . . . I don’t quite understand your question.”

  Autrey and I turn back to our bucket. It’s tim
e to bring in the sheep. You wouldn’t want to forget about that part. Otherwise what would be the point?

  *

  Book II

  How to Actually Change Your Mind

  Rationality: An Introduction

  E. Overly Convenient Excuses

  46. The Proper Use of Humility

  47. The Third Alternative

  48. Lotteries: A Waste of Hope

  49. New Improved Lottery

  50. But There’s Still a Chance, Right?

  51. The Fallacy of Gray

  52. Absolute Authority

  53. How to Convince Me That 2 + 2 = 3

  54. Infinite Certainty

  55. 0 And 1 Are Not Probabilities

  56. Your Rationality Is My Business

  F. Politics and Rationality

  57. Politics is the Mind-Killer

  58. Policy Debates Should Not Appear One-Sided

  59. The Scales of Justice, the Notebook of Rationality

  60. Correspondence Bias

  61. Are Your Enemies Innately Evil?

  62. Reversed Stupidity Is Not Intelligence

  63. Argument Screens Off Authority

  64. Hug the Query

  65. Rationality and the English Language

  66. Human Evil and Muddled Thinking

  G. Against Rationalization

  67. Knowing About Biases Can Hurt People

  68. Update Yourself Incrementally

  69. One Argument Against An Army

  70. The Bottom Line

  71. What Evidence Filtered Evidence?

  72. Rationalization

  73. A Rational Argument

  74. Avoiding Your Belief’s Real Weak Points

  75. Motivated Stopping and Motivated Continuation

  76. Fake Justification

  77. Is That Your True Rejection?

  78. Entangled Truths, Contagious Lies

  79. Of Lies and Black Swan Blowups

  80. Dark Side Epistemology

  H. Against Doublethink

  81. Singlethink

 

‹ Prev