Rationality- From AI to Zombies

by Eliezer Yudkowsky

  You would think, perhaps, that real rationalists ought to be more coordinated? Surely all that unreason must have its disadvantages? That mode can’t be optimal, can it?

  And if current “rationalist” groups cannot coordinate—if they can’t support group projects so well as a single synagogue draws donations from its members—well, I leave it to you to finish that syllogism.

  There’s a saying I sometimes use: “It is dangerous to be half a rationalist.”

  For example, I can think of ways to sabotage someone’s intelligence by selectively teaching them certain methods of rationality. Suppose you taught someone a long list of logical fallacies and cognitive biases, and trained them to spot those fallacies and biases in other people’s arguments. But you are careful to pick those fallacies and biases that are easiest to accuse others of, the most general ones that can easily be misapplied. And you do not warn them to scrutinize arguments they agree with just as hard as they scrutinize incongruent arguments for flaws. So they have acquired a great repertoire of flaws of which to accuse only arguments and arguers who they don’t like. This, I suspect, is one of the primary ways that smart people end up stupid. (And note, by the way, that I have just given you another Fully General Counterargument against smart people whose arguments you don’t like.)

  Similarly, if you wanted to ensure that a group of “rationalists” never accomplished any task requiring more than one person, you could teach them only techniques of individual rationality, without mentioning anything about techniques of coordinated group rationality.

  I’ll write more later on how I think rationalists might be able to coordinate better. But here I want to focus on what you might call the culture of disagreement, or even the culture of objections, which is one of the two major forces preventing the technophile crowd from coordinating.

  Imagine that you’re at a conference, and the speaker gives a thirty-minute talk. Afterward, people line up at the microphones for questions. The first questioner objects to the graph used in slide 14 using a logarithmic scale; they quote Tufte on The Visual Display of Quantitative Information. The second questioner disputes a claim made in slide 3. The third questioner suggests an alternative hypothesis that seems to explain the same data . . .

  Perfectly normal, right? Now imagine that you’re at a conference, and the speaker gives a thirty-minute talk. People line up at the microphone.

  The first person says, “I agree with everything you said in your talk, and I think you’re brilliant.” Then steps aside.

  The second person says, “Slide 14 was beautiful, I learned a lot from it. You’re awesome.” Steps aside.

  The third person—

  Well, you’ll never know what the third person at the microphone had to say, because by this time, you’ve fled screaming out of the room, propelled by a bone-deep terror as if Cthulhu had erupted from the podium, the fear of the impossibly unnatural phenomenon that has invaded your conference.

  Yes, a group that can’t tolerate disagreement is not rational. But if you tolerate only disagreement—if you tolerate disagreement but not agreement—then you also are not rational. You’re only willing to hear some honest thoughts, but not others. You are a dangerous half-a-rationalist.

  We are as uncomfortable together as flying-saucer cult members are uncomfortable apart. That can’t be right either. Reversed stupidity is not intelligence.

  Let’s say we have two groups of soldiers. In group 1, the privates are ignorant of tactics and strategy; only the sergeants know anything about tactics and only the officers know anything about strategy. In group 2, everyone at all levels knows all about tactics and strategy.

  Should we expect group 1 to defeat group 2, because group 1 will follow orders, while everyone in group 2 comes up with better ideas than whatever orders they were given?

  In this case I have to question how much group 2 really understands about military theory, because it is an elementary proposition that an uncoordinated mob gets slaughtered.

  Doing worse with more knowledge means you are doing something very wrong. You should always be able to at least implement the same strategy you would use if you are ignorant, and preferably do better. You definitely should not do worse. If you find yourself regretting your “rationality” then you should reconsider what is rational.

  On the other hand, if you are only half-a-rationalist, you can easily do worse with more knowledge. I recall a lovely experiment which showed that politically opinionated students with more knowledge of the issues reacted less to incongruent evidence, because they had more ammunition with which to counter-argue only incongruent evidence.

  We would seem to be stuck in an awful valley of partial rationality where we end up more poorly coordinated than religious fundamentalists, able to put forth less effort than flying-saucer cultists. True, what little effort we do manage to put forth may be better-targeted at helping people rather than the reverse—but that is not an acceptable excuse.

  If I were setting forth to systematically train rationalists, there would be lessons on how to disagree and lessons on how to agree, lessons intended to make the trainee more comfortable with dissent, and lessons intended to make them more comfortable with conformity. One day everyone shows up dressed differently, another day they all show up in uniform. You’ve got to cover both sides, or you’re only half a rationalist.

  Can you imagine training prospective rationalists to wear a uniform and march in lockstep, and practice sessions where they agree with each other and applaud everything a speaker on a podium says? It sounds like unspeakable horror, doesn’t it, like the whole thing has admitted outright to being an evil cult? But why is it not okay to practice that, while it is okay to practice disagreeing with everyone else in the crowd? Are you never going to have to agree with the majority?

  Our culture puts all the emphasis on heroic disagreement and heroic defiance, and none on heroic agreement or heroic group consensus. We signal our superior intelligence and our membership in the nonconformist community by inventing clever objections to others’ arguments. Perhaps that is why the technophile / Silicon Valley crowd stays marginalized, losing battles with less nonconformist factions in larger society. No, we’re not losing because we’re so superior, we’re losing because our exclusively individualist traditions sabotage our ability to cooperate.

  The other major component that I think sabotages group efforts in the technophile community is being ashamed of strong feelings. We still have the Spock archetype of rationality stuck in our heads, rationality as dispassion. Or perhaps a related mistake, rationality as cynicism—trying to signal your superior world-weary sophistication by showing that you care less than others. Being careful to ostentatiously, publicly look down on those so naive as to show they care strongly about anything.

  Wouldn’t it make you feel uncomfortable if the speaker at the podium said that they cared so strongly about, say, fighting aging, that they would willingly die for the cause?

  But it is nowhere written in either probability theory or decision theory that a rationalist should not care. I’ve looked over those equations and, really, it’s not in there.

  The best informal definition I’ve ever heard of rationality is “That which can be destroyed by the truth should be.” We should aspire to feel the emotions that fit the facts, not aspire to feel no emotion. If an emotion can be destroyed by truth, we should relinquish it. But if a cause is worth striving for, then let us by all means feel fully its importance.

  Some things are worth dying for. Yes, really! And if we can’t get comfortable with admitting it and hearing others say it, then we’re going to have trouble caring enough—as well as coordinating enough—to put some effort into group projects. You’ve got to teach both sides of it, “That which can be destroyed by the truth should be,” and “That which the truth nourishes should thrive.”

  I’ve heard it argued that the taboo against emotional language in, say, science papers, is an important part of letting the facts fight
it out without distraction. That doesn’t mean the taboo should apply everywhere. I think that there are parts of life where we should learn to applaud strong emotional language, eloquence, and poetry. When there’s something that needs doing, poetic appeals help get it done, and, therefore, are themselves to be applauded.

  We need to keep our efforts to expose counterproductive causes and unjustified appeals from stomping on tasks that genuinely need doing. You need both sides of it—the willingness to turn away from counterproductive causes, and the willingness to praise productive ones; the strength to be unswayed by ungrounded appeals, and the strength to be swayed by grounded ones.

  I think the synagogue at their annual appeal had it right, really. They weren’t going down row by row and putting individuals on the spot, staring at them and saying, “How much will you donate, Mr. Schwartz?” People simply announced their pledges—not with grand drama and pride, just simple announcements—and that encouraged others to do the same. Those who had nothing to give, stayed silent; those who had objections, chose some later or earlier time to voice them. That’s probably about the way things should be in a sane human community—taking into account that people often have trouble getting as motivated as they wish they were, and can be helped by social encouragement to overcome this weakness of will.

  But even if you disagree with that part, then let us say that both supporting and countersupporting opinions should have been publicly voiced. Supporters being faced by an apparently solid wall of objections and disagreements—even if it resulted from their own uncomfortable self-censorship—is not group rationality. It is the mere mirror image of what Dark Side groups do to keep their followers. Reversed stupidity is not intelligence.

  *

  318

  Tolerate Tolerance

  One of the likely characteristics of someone who sets out to be a “rationalist” is a lower-than-usual tolerance for flaws in reasoning. This doesn’t strictly follow. You could end up, say, rejecting your religion, just because you spotted more or deeper flaws in the reasoning, not because you were, by your nature, more annoyed at a flaw of fixed size. But realistically speaking, a lot of us probably have our level of “annoyance at all these flaws we’re spotting” set a bit higher than average.

  That’s why it’s so important for us to tolerate others’ tolerance if we want to get anything done together.

  For me, the poster case of tolerance I need to tolerate is Ben Goertzel, who among other things runs an annual AI conference, and who has something nice to say about everyone. Ben even complimented the ideas of M*nt*f*x, the most legendary of all AI crackpots. (M*nt*f*x apparently started adding a link to Ben’s compliment in his email signatures, presumably because it was the only compliment he’d ever gotten from a bona fide AI academic.) (Please do not pronounce his True Name correctly or he will be summoned here.)

  But I’ve come to understand that this is one of Ben’s strengths—that he’s nice to lots of people that others might ignore, including, say, me—and every now and then this pays off for him.

  And if I subtract points off Ben’s reputation for finding something nice to say about people and projects that I think are hopeless—even M*nt*f*x—then what I’m doing is insisting that Ben dislike everyone I dislike before I can work with him.

  Is that a realistic standard? Especially if different people are annoyed in different amounts by different things?

  But it’s hard to remember that when Ben is being nice to so many idiots.

  Cooperation is unstable, in both game theory and evolutionary biology, without some kind of punishment for defection. So it’s one thing to subtract points off someone’s reputation for mistakes they make themselves, directly. But if you also look askance at someone for refusing to castigate a person or idea, then that is punishment of non-punishers, a far more dangerous idiom that can lock an equilibrium in place even if it’s harmful to everyone involved.

  The danger of punishing non-punishers is something I remind myself of, say, every time Robin Hanson points out a flaw in some academic trope and yet modestly confesses he could be wrong (and he’s not wrong). Or every time I see Michael Vassar still considering the potential of someone who I wrote off as hopeless within thirty seconds of being introduced to them. I have to remind myself, “Tolerate tolerance! Don’t demand that your allies be equally extreme in their negative judgments of everything you dislike!”

  By my nature, I do get annoyed when someone else seems to be giving too much credit. I don’t know if everyone’s like that, but I suspect that at least some of my fellow aspiring rationalists are. I wouldn’t be surprised to find it a human universal; it does have an obvious evolutionary rationale—one which would make it a very unpleasant and dangerous adaptation.

  I am not generally a fan of “tolerance.” I certainly don’t believe in being “intolerant of intolerance,” as some inconsistently hold. But I shall go on trying to tolerate people who are more tolerant than I am, and judge them only for their own un-borrowed mistakes.

  Oh, and it goes without saying that if the people of Group X are staring at you demandingly, waiting for you to hate the right enemies with the right intensity, and ready to castigate you if you fail to castigate loudly enough, you may be hanging around the wrong group.

  Just don’t demand that everyone you work with be equally intolerant of behavior like that. Forgive your friends if some of them suggest that maybe Group X wasn’t so awful after all . . .

  *

  319

  Your Price for Joining

  In the Ultimatum Game, the first player chooses how to split $10 between themselves and the second player, and the second player decides whether to accept the split or reject it—in the latter case, both parties get nothing. So far as conventional causal decision theory goes (two-box on Newcomb’s Problem, defect in Prisoner’s Dilemma), the second player should prefer any non-zero amount to nothing. But if the first player expects this behavior—accept any non-zero offer—then they have no motive to offer more than a penny. As I assume you all know by now, I am no fan of conventional causal decision theory. Those of us who remain interested in cooperating on the Prisoner’s Dilemma, either because it’s iterated, or because we have a term in our utility function for fairness, or because we use an unconventional decision theory, may also not accept an offer of one penny.

  And in fact, most Ultimatum “deciders” offer an even split; and most Ultimatum “accepters” reject any offer less than 20%. A 100 USD game played in Indonesia (average per capita income at the time: 670 USD) showed offers of 30 USD being turned down, although this equates to two week’s wages. We can probably also assume that the players in Indonesia were not thinking about the academic debate over Newcomblike problems—this is just the way people feel about Ultimatum Games, even ones played for real money.

  There’s an analogue of the Ultimatum Game in group coordination. (Has it been studied? I’d hope so . . .) Let’s say there’s a common project—in fact, let’s say that it’s an altruistic common project, aimed at helping mugging victims in Canada, or something. If you join this group project, you’ll get more done than you could on your own, relative to your utility function. So, obviously, you should join.

  But wait! The anti-mugging project keeps their funds invested in a money market fund! That’s ridiculous; it won’t earn even as much interest as US Treasuries, let alone a dividend-paying index fund.

  Clearly, this project is run by morons, and you shouldn’t join until they change their malinvesting ways.

  Now you might realize—if you stopped to think about it—that all things considered, you would still do better by working with the common anti-mugging project, than striking out on your own to fight crime. But then—you might perhaps also realize—if you too easily assent to joining the group, why, what motive would they have to change their malinvesting ways?

  Well . . . Okay, look. Possibly because we’re out of the ancestral environment where everyone knows everyone else . . . a
nd possibly because the nonconformist crowd tries to repudiate normal group-cohering forces like conformity and leader-worship . . .

  . . . It seems to me that people in the atheist / libertarian / technophile / science fiction fan / etc. cluster often set their joining prices way way way too high. Like a 50-way split Ultimatum game, where every one of 50 players demands at least 20% of the money.

  If you think how often situations like this would have arisen in the ancestral environment, then it’s almost certainly a matter of evolutionary psychology. System 1 emotions, not System 2 calculation. Our intuitions for when to join groups, versus when to hold out for more concessions to our own preferred way of doing things, would have been honed for hunter-gatherer environments of, e.g., 40 people, all of whom you knew personally.

  And if the group is made up of 1,000 people? Then your hunter-gatherer instincts will underestimate the inertia of a group so large, and demand an unrealistically high price (in strategic shifts) for you to join. There’s a limited amount of organizational effort, and a limited number of degrees of freedom, that can go into doing things any one person’s way.

  And if the strategy is large and complex, the sort of thing that takes e.g. ten people doing paperwork for a week, rather than being hammered out over a half-hour of negotiation around a campfire? Then your hunter-gatherer instincts will underestimate the inertia of the group, relative to your own demands.

  And if you live in a wider world than a single hunter-gatherer tribe, so that you only see the one group representative who negotiates with you, and not the hundred other negotiations that have taken place already? Then your instincts will tell you that it is just one person, a stranger at that, and the two of you are equals; whatever ideas they bring to the table are equal with whatever ideas you bring to the table, and the meeting point ought to be about even.

  And if you suffer from any weakness of will or akrasia, or if you are influenced by motives other than those you would admit to yourself that you are influenced by, then any group-altruistic project that does not offer you the rewards of status and control may perhaps find itself underserved by your attentions.