Rationality- From AI to Zombies

by Eliezer Yudkowsky

  James Thomson’s “A Poem Sacred to the Memory of Sir Isaac Newton,” which praises the rainbow for what it really is—you can argue whether or not Thomson’s poem is as gripping as John Keats’s Lamia who was loved and lost. But tales of love and loss and cynicism had already been told, far away in ancient Greece, and no doubt many times before. Until we understood the rainbow as a thing different from tales of human-shaped magic, the true story of the rainbow could not be poeticized.

  The border between science fiction and space opera was once drawn as follows: If you can take the plot of a story and put it back in the Old West, or the Middle Ages, without changing it, then it is not real science fiction. In real science fiction, the science is intrinsically part of the plot—you can’t move the story from space to the savanna, not without losing something.

  Richard Feynman asked: “What men are poets who can speak of Jupiter if he were like a man, but if he is an immense spinning sphere of methane and ammonia must be silent?”

  They are savanna poets, who can only tell stories that would have made sense around a campfire ten thousand years ago. Savanna poets, who can tell only the Great Stories in their classic forms, and nothing more.

  *

  1. Richard P. Feynman, Robert B. Leighton, and Matthew L. Sands, The Feynman Lectures on Physics, 3 vols. (Reading, MA: Addison-Wesley, 1963).

  202

  Joy in the Merely Real

  . . . Do not all charms fly

  At the mere touch of cold philosophy?

  There was an awful rainbow once in heaven:

  We know her woof, her texture; she is given

  In the dull catalogue of common things.

  —John Keats, Lamia

  Nothing is “mere.”

  —Richard Feynman

  You’ve got to admire that phrase, “dull catalogue of common things.” What is it, exactly, that goes in this catalogue? Besides rainbows, that is?

  Why, things that are mundane, of course. Things that are normal; things that are unmagical; things that are known, or knowable; things that play by the rules (or that play by any rules, which makes them boring); things that are part of the ordinary universe; things that are, in a word, real.

  Now that’s what I call setting yourself up for a fall.

  At that rate, sooner or later you’re going to be disappointed in everything—either it will turn out not to exist, or even worse, it will turn out to be real.

  If we cannot take joy in things that are merely real, our lives will always be empty.

  For what sin are rainbows demoted to the dull catalogue of common things? For the sin of having a scientific explanation. “We know her woof, her texture,” says Keats—an interesting use of the word “we,” because I suspect that Keats didn’t know the explanation himself. I suspect that just being told that someone else knew was too much for him to take. I suspect that just the notion of rainbows being scientifically explicable in principle would have been too much to take. And if Keats didn’t think like that, well, I know plenty of people who do.

  I have already remarked that nothing is inherently mysterious—nothing that actually exists, that is. If I am ignorant about a phenomenon, that is a fact about my state of mind, not a fact about the phenomenon; to worship a phenomenon because it seems so wonderfully mysterious is to worship your own ignorance; a blank map does not correspond to a blank territory, it is just somewhere we haven’t visited yet, etc., etc. . . .

  Which is to say that everything—everything that actually exists—is liable to end up in “the dull catalogue of common things,” sooner or later.

  Your choice is either:

  Decide that things are allowed to be unmagical, knowable, scientifically explicable—in a word, real—and yet still worth caring about;

  Or go about the rest of your life suffering from existential ennui that is unresolvable.

  (Self-deception might be an option for others, but not for you.)

  This puts quite a different complexion on the bizarre habit indulged by those strange folk called scientists, wherein they suddenly become fascinated by pocket lint or bird droppings or rainbows, or some other ordinary thing which world-weary and sophisticated folk would never give a second glance.

  You might say that scientists—at least some scientists—are those folk who are in principle capable of enjoying life in the real universe.

  *

  Part Q

  Joy in the Merely Real

  203

  Joy in Discovery

  Newton was the greatest genius who ever lived, and the most fortunate; for we cannot find more than once a system of the world to establish.

  —Lagrange

  I have more fun discovering things for myself than reading about them in textbooks. This is right and proper, and only to be expected.

  But discovering something that no one else knows—being the first to unravel the secret—

  There is a story that one of the first men to realize that stars were burning by fusion—plausible attributions I’ve seen are to Fritz Houtermans and Hans Bethe—was walking out with his girlfriend of a night, and she made a comment on how beautiful the stars were, and he replied: “Yes, and right now, I’m the only man in the world who knows why they shine.”

  It is attested by numerous sources that this experience, being the first person to solve a major mystery, is a tremendous high. It’s probably the closest experience you can get to taking drugs, without taking drugs—though I wouldn’t know.

  That can’t be healthy.

  Not that I’m objecting to the euphoria. It’s the exclusivity clause that bothers me. Why should a discovery be worth less, just because someone else already knows the answer?

  The most charitable interpretation I can put on the psychology, is that you don’t struggle with a single problem for months or years if it’s something you can just look up in the library. And that the tremendous high comes from having hit the problem from every angle you can manage, and having bounced; and then having analyzed the problem again, using every idea you can think of, and all the data you can get your hands on—making progress a little at a time—so that when, finally, you crack through the problem, all the dangling pieces and unresolved questions fall into place at once, like solving a dozen locked-room murder mysteries with a single clue.

  And more, the understanding you get is real understanding—understanding that embraces all the clues you studied to solve the problem, when you didn’t yet know the answer. Understanding that comes from asking questions day after day and worrying at them; understanding that no one else can get (no matter how much you tell them the answer) unless they spend months studying the problem in its historical context, even after it’s been solved—and even then, they won’t get the high of solving it all at once.

  That’s one possible reason why James Clerk Maxwell might have had more fun discovering Maxwell’s equations, than you had fun reading about them.

  A slightly less charitable reading is that the tremendous high comes from what is termed, in the politesse of social psychology, “commitment” and “consistency” and “cognitive dissonance”; the part where we value something more highly just because it took more work to get it. The studies showing that subjecting fraternity pledges to a harsher initiation, causes them to be more convinced of the value of the fraternity—identical wine in higher-priced bottles being rated as tasting better—that sort of thing.

  Of course, if you just have more fun solving a puzzle than being told its answer, because you enjoy doing the cognitive work for its own sake, there’s nothing wrong with that. The less charitable reading would be if charging $100 to be told the answer to a puzzle made you think the answer was more interesting, worthwhile, important, surprising, etc., than if you got the answer for free.

  (I strongly suspect that a major part of science’s PR problem in the population at large is people who instinctively believe that if knowledge is given away for free, it cannot be important. If you had to undergo a fe
arsome initiation ritual to be told the truth about evolution, maybe people would be more satisfied with the answer.)

  The really uncharitable reading is that the joy of first discovery is about status. Competition. Scarcity. Beating everyone else to the punch. It doesn’t matter whether you have a three-room house or a four-room house, what matters is having a bigger house than the Joneses. A two-room house would be fine, if you could only ensure that the Joneses had even less.

  I don’t object to competition as a matter of principle. I don’t think that the game of Go is barbaric and should be suppressed, even though it’s zero-sum. But if the euphoric joy of scientific discovery has to be about scarcity, that means it’s only available to one person per civilization for any given truth.

  If the joy of scientific discovery is one-shot per discovery, then, from a fun-theoretic perspective, Newton probably used up a substantial increment of the total Physics Fun available over the entire history of Earth-originating intelligent life. That selfish bastard explained the orbits of planets and the tides.

  And really the situation is even worse than this, because in the Standard Model of physics (discovered by bastards who spoiled the puzzle for everyone else) the universe is spatially infinite, inflationarily branching, and branching via decoherence, which is at least three different ways that Reality is exponentially or infinitely large.

  So aliens, or alternate Newtons, or just Tegmark duplicates of Newton, may all have discovered gravity before our Newton did—if you believe that “before” means anything relative to those kinds of separations.

  When that thought first occurred to me, I actually found it quite uplifting. Once I realized that someone, somewhere in the expanses of space and time, already knows the answer to any answerable question—even biology questions and history questions; there are other decoherent Earths—then I realized how silly it was to think as if the joy of discovery ought to be limited to one person. It becomes a fully inescapable source of unresolvable existential angst, and I regard that as a reductio.

  The consistent solution which maintains the possibility of fun is to stop worrying about what other people know. If you don’t know the answer, it’s a mystery to you. If you can raise your hand, and clench your fingers into a fist, and you’ve got no idea of how your brain is doing it—or even what exact muscles lay beneath your skin—you’ve got to consider yourself just as ignorant as a hunter-gatherer. Sure, someone else knows the answer—but back in the hunter-gatherer days, someone else in an alternate Earth, or for that matter, someone else in the future, knew what the answer was. Mystery, and the joy of finding out, is either a personal thing, or it doesn’t exist at all—and I prefer to say it’s personal.

  The joy of assisting your civilization by telling it something it doesn’t already know does tend to be one-shot per discovery per civilization; that kind of value is conserved, as are Nobel Prizes. And the prospect of that reward may be what it takes to keep you focused on one problem for the years required to develop a really deep understanding; plus, working on a problem unknown to your civilization is a sure-fire way to avoid reading any spoilers.

  But as part of my general project to undo this idea that rationalists have less fun, I want to restore the magic and mystery to every part of the world that you do not personally understand, regardless of what other knowledge may exist, far away in space and time, or even in your next-door neighbor’s mind. If you don’t know, it’s a mystery. And now think of how many things you don’t know! (If you can’t think of anything, you have other problems.) Isn’t the world suddenly a much more mysterious and magical and interesting place? As if you’d been transported into an alternate dimension, and had to learn all the rules from scratch?

  A friend once told me that I look at the world as if I’ve never seen it before. I thought, that’s a nice compliment . . . Wait! I never have seen it before! What—did everyone else get a preview?

  —Ran Prieur

  *

  204

  Bind Yourself to Reality

  So perhaps you’re reading all this, and asking: “Yes, but what does this have to do with reductionism?”

  Partially, it’s a matter of leaving a line of retreat. It’s not easy to take something important apart into components, when you’re convinced that this removes magic from the world, unweaves the rainbow. I do plan to take certain things apart, in this book; and I prefer not to create pointless existential anguish.

  Partially, it’s the crusade against Hollywood Rationality, the concept that understanding the rainbow subtracts its beauty. The rainbow is still beautiful plus you get the beauty of physics.

  But even more deeply, it’s one of these subtle hidden-core-of-rationality things. You know, the sort of thing where I start talking about “the Way.” It’s about binding yourself to reality.

  In one of Frank Herbert’s Dune books, if I recall correctly, it is said that a Truthsayer gains their ability to detect lies in others by always speaking truth themselves, so that they form a relationship with the truth whose violation they can feel. It wouldn’t work, but I still think it’s one of the more beautiful thoughts in fiction. At the very least, to get close to the truth, you have to be willing to press yourself up against reality as tightly as possible, without flinching away, or sneering down.

  You can see the bind-yourself-to-reality theme in Lotteries: A Waste of Hope. Understanding that lottery tickets have negative expected utility does not mean that you give up the hope of being rich. It means that you stop wasting that hope on lottery tickets. You put the hope into your job, your school, your startup, your eBay sideline; and if you truly have nothing worth hoping for, then maybe it’s time to start looking.

  It’s not dreams I object to, only impossible dreams. The lottery isn’t impossible, but it is an un-actionable near-impossibility. It’s not that winning the lottery is extremely difficult—requires a desperate effort—but that work isn’t the issue.

  I say all this, to exemplify the idea of taking emotional energy that is flowing off to nowhere, and binding it into the realms of reality.

  This doesn’t mean setting goals that are low enough to be “realistic,” i.e., easy and safe and parentally approved. Maybe this is good advice in your personal case, I don’t know, but I’m not the one to say it.

  What I mean is that you can invest emotional energy in rainbows even if they turn out not to be magic. The future is always absurd but it is never unreal.

  The Hollywood Rationality stereotype is that “rational = emotionless”; the more reasonable you are, the more of your emotions Reason inevitably destroys. In Feeling Rational I contrast this against “That which can be destroyed by the truth should be” and “That which the truth nourishes should thrive.” When you have arrived at your best picture of the truth, there is nothing irrational about the emotions you feel as a result of that—the emotions cannot be destroyed by truth, so they must not be irrational.

  So instead of destroying emotional energies associated with bad explanations for rainbows, as the Hollywood Rationality stereotype would have it, let us redirect these emotional energies into reality—bind them to beliefs that are as true as we can make them.

  Want to fly? Don’t give up on flight. Give up on flying potions and build yourself an airplane.

  Remember the theme of Think like Reality, where I talked about how when physics seems counterintuitive, you’ve got to accept that it’s not physics that’s weird, it’s you?

  What I’m talking about now is like that, only with emotions instead of hypotheses—binding your feelings into the real world. Not the “realistic” everyday world. I would be a howling hypocrite if I told you to shut up and do your homework. I mean the real real world, the lawful universe, that includes absurdities like Moon landings and the evolution of human intelligence. Just not any magic, anywhere, ever.

  It is a Hollywood Rationality meme that “Science takes the fun out of life.”

  Science puts the fun back into life.

>   Rationality directs your emotional energies into the universe, rather than somewhere else.

  *

  205

  If You Demand Magic, Magic Won’t Help

  Most witches don’t believe in gods. They know that the gods exist, of course. They even deal with them occasionally. But they don’t believe in them. They know them too well. It would be like believing in the postman.

  —Terry Pratchett, Witches Abroad1

  Once upon a time, I was pondering the philosophy of fantasy stories—

  And before anyone chides me for my “failure to understand what fantasy is about,” let me say this: I was raised in a science fiction and fantasy household. I have been reading fantasy stories since I was five years old. I occasionally try to write fantasy stories. And I am not the sort of person who tries to write for a genre without pondering its philosophy. Where do you think story ideas come from?

  Anyway:

  I was pondering the philosophy of fantasy stories, and it occurred to me that if there were actually dragons in our world—if you could go down to the zoo, or even to a distant mountain, and meet a fire-breathing dragon—while nobody had ever actually seen a zebra, then our fantasy stories would contain zebras aplenty, while dragons would be unexciting.

  Now that’s what I call painting yourself into a corner, wot? The grass is always greener on the other side of unreality.

  In one of the standard fantasy plots, a protagonist from our Earth, a sympathetic character with lousy grades or a crushing mortgage but still a good heart, suddenly finds themselves in a world where magic operates in place of science. The protagonist often goes on to practice magic, and become in due course a (superpowerful) sorcerer.