Rationality- From AI to Zombies

by Eliezer Yudkowsky

  Sneaking in Connotations

  In the previous essay, we saw that in Japan, blood types have taken the place of astrology—if your blood type is AB, for example, you’re supposed to be “cool and controlled.”

  So suppose we decided to invent a new word, “wiggin,” and defined this word to mean people with green eyes and black hair—

  A green-eyed man with black hair walked into a restaurant.

  “Ha,” said Danny, watching from a nearby table, “did you see that? A wiggin just walked into the room. Bloody wiggins. Commit all sorts of crimes, they do.”

  His sister Erda sighed. “You haven’t seen him commit any crimes, have you, Danny?”

  “Don’t need to,” Danny said, producing a dictionary. “See, it says right here in the Oxford English Dictionary. ‘Wiggin. (1) A person with green eyes and black hair.’ He’s got green eyes and black hair, he’s a wiggin. You’re not going to argue with the Oxford English Dictionary, are you? By definition, a green-eyed black-haired person is a wiggin.”

  “But you called him a wiggin,” said Erda. “That’s a nasty thing to say about someone you don’t even know. You’ve got no evidence that he puts too much ketchup on his burgers, or that as a kid he used his slingshot to launch baby squirrels.”

  “But he is a wiggin,” Danny said patiently. “He’s got green eyes and black hair, right? Just you watch, as soon as his burger arrives, he’s reaching for the ketchup.”

  The human mind passes from observed characteristics to inferred characteristics via the medium of words. In “All humans are mortal, Socrates is a human, therefore Socrates is mortal,” the observed characteristics are Socrates’s clothes, speech, tool use, and generally human shape; the categorization is “human”; the inferred characteristic is poisonability by hemlock.

  Of course there’s no hard distinction between “observed characteristics” and “inferred characteristics.” If you hear someone speak, they’re probably shaped like a human, all else being equal. If you see a human figure in the shadows, then ceteris paribus it can probably speak.

  And yet some properties do tend to be more inferred than observed. You’re more likely to decide that someone is human, and will therefore burn if exposed to open flame, than carry through the inference the other way around.

  If you look in a dictionary for the definition of “human,” you’re more likely to find characteristics like “intelligence” and “featherless biped”—characteristics that are useful for quickly eyeballing what is and isn’t a human—rather than the ten thousand connotations, from vulnerability to hemlock, to overconfidence, that we can infer from someone’s being human. Why? Perhaps dictionaries are intended to let you match up labels to similarity groups, and so are designed to quickly isolate clusters in thingspace. Or perhaps the big, distinguishing characteristics are the most salient, and therefore first to pop into a dictionary editor’s mind. (I’m not sure how aware dictionary editors are of what they really do.)

  But the upshot is that when Danny pulls out his OED to look up “wiggin,” he sees listed only the first-glance characteristics that distinguish a wiggin: Green eyes and black hair. The OED doesn’t list the many minor connotations that have come to attach to this term, such as criminal proclivities, culinary peculiarities, and some unfortunate childhood activities.

  How did those connotations get there in the first place? Maybe there was once a famous wiggin with those properties. Or maybe someone made stuff up at random, and wrote a series of bestselling books about it (The Wiggin, Talking to Wiggins, Raising Your Little Wiggin, Wiggins in the Bedroom). Maybe even the wiggins believe it now, and act accordingly. As soon as you call some people “wiggins,” the word will begin acquiring connotations.

  But remember the Parable of Hemlock: If we go by the logical class definitions, we can never class Socrates as a “human” until after we observe him to be mortal. Whenever someone pulls a dictionary, they’re generally trying to sneak in a connotation, not the actual definition written down in the dictionary.

  After all, if the only meaning of the word “wiggin” is “green-eyed black-haired person,” then why not just call those people “green-eyed black-haired people”? And if you’re wondering whether someone is a ketchup-reacher, why not ask directly, “Is he a ketchup-reacher?” rather than “Is he a wiggin?” (Note substitution of substance for symbol.)

  Oh, but arguing the real question would require work. You’d have to actually watch the wiggin to see if he reached for the ketchup. Or maybe see if you can find statistics on how many green-eyed black-haired people actually like ketchup. At any rate, you wouldn’t be able to do it sitting in your living room with your eyes closed. And people are lazy. They’d rather argue “by definition,” especially since they think “you can define a word any way you like.”

  But of course the real reason they care whether someone is a “wiggin” is a connotation—a feeling that comes along with the word—that isn’t in the definition they claim to use.

  Imagine Danny saying, “Look, he’s got green eyes and black hair. He’s a wiggin! It says so right there in the dictionary!—therefore, he’s got black hair. Argue with that, if you can!”

  Doesn’t have much of a triumphant ring to it, does it? If the real point of the argument actually was contained in the dictionary definition—if the argument genuinely was logically valid—then the argument would feel empty; it would either say nothing new, or beg the question.

  It’s only the attempt to smuggle in connotations not explicitly listed in the definition, that makes anyone feel they can score a point that way.

  *

  172

  Arguing “By Definition”

  “This plucked chicken has two legs and no feathers—therefore, by definition, it is a human!”

  When people argue definitions, they usually start with some visible, known, or at least widely believed set of characteristics; then pull out a dictionary, and point out that these characteristics fit the dictionary definition; and so conclude, “Therefore, by definition, atheism is a religion!”

  But visible, known, widely believed characteristics are rarely the real point of a dispute. Just the fact that someone thinks Socrates’s two legs are evident enough to make a good premise for the argument, “Therefore, by definition, Socrates is human!” indicates that bipedalism probably isn’t really what’s at stake—or the listener would reply, “Whaddaya mean Socrates is bipedal? That’s what we’re arguing about in the first place!”

  Now there is an important sense in which we can legitimately move from evident characteristics to not-so-evident ones. You can, legitimately, see that Socrates is human-shaped, and predict his vulnerability to hemlock. But this probabilistic inference does not rely on dictionary definitions or common usage; it relies on the universe containing empirical clusters of similar things.

  This cluster structure is not going to change depending on how you define your words. Even if you look up the dictionary definition of “human” and it says “all featherless bipeds except Socrates,” that isn’t going to change the actual degree to which Socrates is similar to the rest of us featherless bipeds.

  When you are arguing correctly from cluster structure, you’ll say something like, “Socrates has two arms, two feet, a nose and tongue, speaks fluent Greek, uses tools, and in every aspect I’ve been able to observe him, seems to have every major and minor property that characterizes Homo sapiens; so I’m going to guess that he has human DNA, human biochemistry, and is vulnerable to hemlock just like all other Homo sapiens in whom hemlock has been clinically tested for lethality.”

  And suppose I reply, “But I saw Socrates out in the fields with some herbologists; I think they were trying to prepare an antidote. Therefore I don’t expect Socrates to keel over after he drinks the hemlock—he will be an exception to the general behavior of objects in his cluster: they did not take an antidote, and he did.”

  Now there’s not much point in arguing over whether Socrates is “h
uman” or not. The conversation has to move to a more detailed level, poke around inside the details that make up the “human” category—talk about human biochemistry, and specifically, the neurotoxic effects of coniine.

  If you go on insisting, “But Socrates is a human and humans, by definition, are mortal!” then what you’re really trying to do is blur out everything you know about Socrates except the fact of his humanity—insist that the only correct prediction is the one you would make if you knew nothing about Socrates except that he was human.

  Which is like insisting that a coin is 50% likely to be showing heads or tails, because it is a “fair coin,” after you’ve actually looked at the coin and it’s showing heads. It’s like insisting that Frodo has ten fingers, because most hobbits have ten fingers, after you’ve already looked at his hands and seen nine fingers. Naturally this is illegal under Bayesian probability theory: You can’t just refuse to condition on new evidence.

  And you can’t just keep one categorization and make estimates based on that, while deliberately throwing out everything else you know.

  Not every piece of new evidence makes a significant difference, of course. If I see that Socrates has nine fingers, this isn’t going to noticeably change my estimate of his vulnerability to hemlock, because I’ll expect that the way Socrates lost his finger didn’t change the rest of his biochemistry. And this is true, whether or not the dictionary’s definition says that human beings have ten fingers. The legal inference is based on the cluster structure of the environment, and the causal structure of biology; not what the dictionary editor writes down, nor even “common usage.”

  Now ordinarily, when you’re doing this right—in a legitimate way—you just say, “The coniine alkaloid found in hemlock produces muscular paralysis in humans, resulting in death by asphyxiation.” Or more simply, “Humans are vulnerable to hemlock.” That’s how it’s usually said in a legitimate argument.

  When would someone feel the need to strengthen the argument with the emphatic phrase “by definition”? (E.g. “Humans are vulnerable to hemlock by definition!”) Why, when the inferred characteristic has been called into doubt—Socrates has been seen consulting herbologists—and so the speaker feels the need to tighten the vise of logic.

  So when you see “by definition” used like this, it usually means: “Forget what you’ve heard about Socrates consulting herbologists—humans, by definition, are mortal!”

  People feel the need to squeeze the argument onto a single course by saying “Any P, by definition, has property Q!,” on exactly those occasions when they see, and prefer to dismiss out of hand, additional arguments that call into doubt the default inference based on clustering.

  So too with the argument “X, by definition, is a Y!” E.g., “Atheists believe that God doesn’t exist; therefore atheists have beliefs about God, because a negative belief is still a belief; therefore atheism asserts answers to theological questions; therefore atheism is, by definition, a religion.”

  You wouldn’t feel the need to say, “Hinduism, by definition, is a religion!” because, well, of course Hinduism is a religion. It’s not just a religion “by definition,” it’s, like, an actual religion.

  Atheism does not resemble the central members of the “religion” cluster, so if it wasn’t for the fact that atheism is a religion by definition, you might go around thinking that atheism wasn’t a religion. That’s why you’ve got to crush all opposition by pointing out that “Atheism is a religion” is true by definition, because it isn’t true any other way.

  Which is to say: People insist that “X, by definition, is a Y!” on those occasions when they’re trying to sneak in a connotation of Y that isn’t directly in the definition, and X doesn’t look all that much like other members of the Y cluster.

  Over the last thirteen years I’ve been keeping track of how often this phrase is used correctly versus incorrectly—though not with literal statistics, I fear. But eyeballing suggests that using the phrase by definition, anywhere outside of math, is among the most alarming signals of flawed argument I’ve ever found. It’s right up there with “Hitler,” “God,” “absolutely certain,” and “can’t prove that.”

  This heuristic of failure is not perfect—the first time I ever spotted a correct usage outside of math, it was by Richard Feynman; and since then I’ve spotted more. But you’re probably better off just deleting the phrase “by definition” from your vocabulary—and always on any occasion where you might be tempted to say it in italics or followed with an exclamation mark. That’s a bad idea by definition!

  *

  173

  Where to Draw the Boundary?

  The one comes to you and says:

  Long have I pondered the meaning of the word “Art,” and at last I’ve found what seems to me a satisfactory definition: “Art is that which is designed for the purpose of creating a reaction in an audience.”

  Just because there’s a word “art” doesn’t mean that it has a meaning, floating out there in the void, which you can discover by finding the right definition.

  It feels that way, but it is not so.

  Wondering how to define a word means you’re looking at the problem the wrong way—searching for the mysterious essence of what is, in fact, a communication signal.

  Now, there is a real challenge which a rationalist may legitimately attack, but the challenge is not to find a satisfactory definition of a word. The real challenge can be played as a single-player game, without speaking aloud. The challenge is figuring out which things are similar to each other—which things are clustered together—and sometimes, which things have a common cause.

  If you define “eluctromugnetism” to include lightning, include compasses, exclude light, and include Mesmer’s “animal magnetism” (what we now call hypnosis), then you will have some trouble asking “How does eluctromugnetism work?” You have lumped together things which do not belong together, and excluded others that would be needed to complete a set. (This example is historically plausible; Mesmer came before Faraday.)

  We could say that eluctromugnetism is a wrong word, a boundary in thingspace that loops around and swerves through the clusters, a cut that fails to carve reality along its natural joints.

  Figuring where to cut reality in order to carve along the joints—this is the problem worthy of a rationalist. It is what people should be trying to do, when they set out in search of the floating essence of a word.

  And make no mistake: it is a scientific challenge to realize that you need a single word to describe breathing and fire. So do not think to consult the dictionary editors, for that is not their job.

  What is “art”? But there is no essence of the word, floating in the void.

  Perhaps you come to me with a long list of the things that you call “art” and “not art”:

  The Little Fugue in G Minor: Art.

  A punch in the nose: Not art.

  Escher’s Relativity: Art.

  A flower: Not art.

  The Python programming language: Art.

  A cross floating in urine: Not art.

  Jack Vance’s Tschai novels: Art.

  Modern Art: Not art.

  And you say to me: “It feels intuitive to me to draw this boundary, but I don’t know why—can you find me an intension that matches this extension? Can you give me a simple description of this boundary?”

  So I reply: “I think it has to do with admiration of craftsmanship: work going in and wonder coming out. What the included items have in common is the similar aesthetic emotions that they inspire, and the deliberate human effort that went into them with the intent of producing such an emotion.”

  Is this helpful, or is it just cheating at Taboo? I would argue that the list of which human emotions are or are not aesthetic is far more compact than the list of everything that is or isn’t art. You might be able to see those emotions lighting up an fMRI scan—I say this by way of emphasizing that emotions are not ethereal.

  But of
course my definition of art is not the real point. The real point is that you could well dispute either the intension or the extension of my definition.

  You could say, “Aesthetic emotion is not what these things have in common; what they have in common is an intent to inspire any complex emotion for the sake of inspiring it.” That would be disputing my intension, my attempt to draw a curve through the data points. You would say, “Your equation may roughly fit those points, but it is not the true generating distribution.”

  Or you could dispute my extension by saying, “Some of these things do belong together—I can see what you’re getting at—but the Python language shouldn’t be on the list, and Modern Art should be.” (This would mark you as a philistine, but you could argue it.) Here, the presumption is that there is indeed an underlying curve that generates this apparent list of similar and dissimilar things—that there is a rhyme and reason, even though you haven’t said yet where it comes from—but I have unwittingly lost the rhythm and included some data points from a different generator.

  Long before you know what it is that electricity and magnetism have in common, you might still suspect—based on surface appearances—that “animal magnetism” does not belong on the list.

  Once upon a time it was thought that the word “fish” included dolphins. Now you could play the oh-so-clever arguer, and say, “The list: {Salmon, guppies, sharks, dolphins, trout} is just a list—you can’t say that a list is wrong. I can prove in set theory that this list exists. So my definition of fish, which is simply this extensional list, cannot possibly be ‘wrong’ as you claim.”

  Or you could stop playing games and admit that dolphins don’t belong on the fish list.

  You come up with a list of things that feel similar, and take a guess at why this is so. But when you finally discover what they really have in common, it may turn out that your guess was wrong. It may even turn out that your list was wrong.