Rationality- From AI to Zombies

by Eliezer Yudkowsky

  And then things got all civilized. And so things went downhill to the point that we have videos on Youtube of supposed Nth-dan black belts being pounded into the ground by someone with real fighting experience.

  I heard of one case of this that was really sad; it was a master of a school who was convinced he could use ki techniques. His students would actually fall over when he used ki attacks, a strange and remarkable and frightening case of self-hypnosis or something . . . and the master goes up against a skeptic and of course gets pounded completely into the floor.

  Truly is it said that “how to not lose” is more broadly applicable information than “how to win.” Every single one of these risk factors transfers straight over to any attempt to start a “rationality dojo.” I put to you the question: What can be done about it?

  *

  1. Gillian Russell, “Epistemic Viciousness in the Martial Arts,” in Martial Arts and Philosophy: Beating and Nothingness, ed. Graham Priest and Damon A. Young (Open Court, 2010).

  315

  Schools Proliferating Without Evidence

  Robyn Dawes, author of one of the original papers from Judgment Under Uncertainty and of the book Rational Choice in an Uncertain World—one of the few who tries really hard to import the results to real life—is also the author of House of Cards: Psychology and Psychotherapy Built on Myth.

  From House of Cards, chapter 1:1

  The ability of these professionals has been subjected to empirical scrutiny—for example, their effectiveness as therapists (Chapter 2), their insight about people (Chapter 3), and the relationship between how well they function and the amount of experience they have had in their field (Chapter 4). Virtually all the research—and this book will reference more than three hundred empirical investigations and summaries of investigations—has found that these professionals’ claims to superior intuitive insight, understanding, and skill as therapists are simply invalid . . .

  Remember Rorschach ink-blot tests? It’s such an appealing argument: the patient looks at the ink-blot and says what they see, the psychotherapist interprets their psychological state based on this. There’ve been hundreds of experiments looking for some evidence that it actually works. Since you’re reading this, you can guess the answer is simply “No.” Yet the Rorschach is still in use. It’s just such a good story that psychotherapists simply can’t bring themselves to believe the vast mounds of experimental evidence saying it doesn’t work—

  —which tells you what sort of field we’re dealing with here.

  And the experimental results on the field as a whole are commensurate. Yes, patients who see psychotherapists have been known to get better faster than patients who simply do nothing. But there is no statistically discernible difference between the many schools of psychotherapy. There is no discernible gain from years of expertise.

  And there’s also no discernible difference between seeing a psychotherapist and spending the same amount of time talking to a randomly selected college professor from another field. It’s just talking to anyone that helps you get better, apparently.

  In the entire absence of the slightest experimental evidence for their effectiveness, psychotherapists became licensed by states, their testimony accepted in court, their teaching schools accredited, and their bills paid by health insurance.

  And there was also a huge proliferation of “schools,” of traditions of practice, in psychotherapy; despite—or perhaps because of—the lack of any experiments showing that one school was better than another . . .

  I should really write more some other time on all the sad things this says about our world. About how the essence of medicine, as recognized by society and the courts, is not a repertoire of procedures with statistical evidence for their healing effectiveness; but, rather, the right air of authority.

  But the subject here is the proliferation of traditions in psychotherapy. So far as I can discern, this was the way you picked up prestige in the field—not by discovering an amazing new technique whose effectiveness could be experimentally verified and adopted by all; but, rather, by splitting off your own “school,” supported by your charisma as founder, and by the good stories you told about all the reasons your techniques should work.

  This was probably, to no small extent, responsible for the existence and continuation of psychotherapy in the first place—the promise of making yourself a Master, like Freud who’d done it first (also without the slightest scrap of experimental evidence). That’s the brass ring of success to chase—the prospect of being a guru and having your own adherents. It’s the struggle for adherents that keeps the clergy vital.

  That’s what happens to a field when it unbinds itself from the experimental evidence—though there were other factors that also placed psychotherapists at risk, such as the deference shown them by their patients, the wish of society to believe that mental healing was possible, and, of course, the general dangers of telling people how to think.

  (Dawes wrote in the ’80s and I know that the Rorschach was still in use as recently as the ’90s, but it’s possible matters have improved since then (as one commenter states). I do remember hearing that there was positive evidence for the greater effectiveness of cognitive-behavioral therapy.)

  The field of hedonic psychology (happiness studies) began, to some extent, with the realization that you could measure happiness—that there was a family of measures that by golly did validate well against each other.

  The act of creating a new measurement creates new science; if it’s a good measurement, you get good science.

  If you’re going to create an organized practice of anything, you really do need some way of telling how well you’re doing, and a practice of doing serious testing—that means a control group, an experimental group, and statistics—on plausible-sounding techniques that people come up with. You really need it.

  *

  1. Robyn M. Dawes, House of Cards: Psychology and Psychotherapy Built on Myth (Free Press, 1996).

  316

  Three Levels of Rationality Verification

  I strongly suspect that there is a possible art of rationality (attaining the map that reflects the territory, choosing so as to direct reality into regions high in your preference ordering) that goes beyond the skills that are standard, and beyond what any single practitioner singly knows. I have a sense that more is possible.

  The degree to which a group of people can do anything useful about this, will depend overwhelmingly on what methods we can devise to verify our many amazing good ideas.

  I suggest stratifying verification methods into three levels of usefulness:

  Reputational

  Experimental

  Organizational.

  If your martial arts master occasionally fights realistic duels (ideally, real duels) against the masters of other schools, and wins or at least doesn’t lose too often, then you know that the master’s reputation is grounded in reality; you know that your master is not a complete poseur. The same would go if your school regularly competed against other schools. You’d be keepin’ it real.

  Some martial arts fail to compete realistically enough, and their students go down in seconds against real streetfighters. Other martial arts schools fail to compete at all—except based on charisma and good stories—and their masters decide they have chi powers. In this latter class we can also place the splintered schools of psychoanalysis.

  So even just the basic step of trying to ground reputations in some realistic trial other than charisma and good stories has tremendous positive effects on a whole field of endeavor.

  But that doesn’t yet get you a science. A science requires that you be able to test 100 applications of method A against 100 applications of method B and run statistics on the results. Experiments have to be replicable and replicated. This requires standard measurements that can be run on students who’ve been taught using randomly-assigned alternative methods, not just realistic duels fought between masters using all of their accumulated techniques
and strength.

  The field of happiness studies was created, more or less, by realizing that asking people “On a scale of 1 to 10, how good do you feel right now?” was a measure that statistically validated well against other ideas for measuring happiness. And this, despite all skepticism, looks like it’s actually a pretty useful measure of some things, if you ask 100 people and average the results.

  But suppose you wanted to put happier people in positions of power—pay happy people to train other people to be happier, or employ the happiest at a hedge fund? Then you’re going to need some test that’s harder to game than just asking someone “How happy are you?”

  This question of verification methods good enough to build organizations is a huge problem at all levels of modern human society. If you’re going to use the SAT to control admissions to elite colleges, then can the SAT be defeated by studying just for the SAT in a way that ends up not correlating to other scholastic potential? If you give colleges the power to grant degrees, then do they have an incentive not to fail people? (I consider it drop-dead obvious that the task of verifying acquired skills and hence the power to grant degrees should be separated from the institutions that do the teaching, but let’s not go into that.) If a hedge fund posts 20% returns, are they really that much better than the indices, or are they selling puts that will blow up in a down market?

  If you have a verification method that can be gamed, the whole field adapts to game it, and loses its purpose. Colleges turn into tests of whether you can endure the classes. High schools do nothing but teach to statewide tests. Hedge funds sell puts to boost their returns.

  On the other hand—we still manage to teach engineers, even though our organizational verification methods aren’t perfect. So what perfect or imperfect methods could you use for verifying rationality skills, that would be at least a little resistant to gaming?

  (Measurements with high noise can still be used experimentally, if you randomly assign enough subjects to have an expectation of washing out the variance. But for the organizational purpose of verifying particular individuals, you need low-noise measurements.)

  So I now put to you the question—how do you verify rationality skills? At any of the three levels? Brainstorm, I beg you; even a difficult and expensive measurement can become a gold standard to verify other metrics. Feel free to email me at yudkowsky@gmail.com to suggest any measurements that are better off not being publicly known (though this is of course a major disadvantage of that method). Stupid ideas can suggest good ideas, so if you can’t come up with a good idea, come up with a stupid one.

  Reputational, experimental, organizational:

  Something the masters and schools can do to keep it real (realistically real);

  Something you can do to measure each of a hundred students;

  Something you could use as a test even if people have an incentive to game it.

  Finding good solutions at each level determines what a whole field of study can be useful for—how much it can hope to accomplish. This is one of the Big Important Foundational Questions, so—

  Think!

  (PS: And ponder on your own before you look at others’ ideas; we need breadth of coverage here.)

  *

  317

  Why Our Kind Can’t Cooperate

  From when I was still forced to attend, I remember our synagogue’s annual fundraising appeal. It was a simple enough format, if I recall correctly. The rabbi and the treasurer talked about the shul’s expenses and how vital this annual fundraise was, and then the synagogue’s members called out their pledges from their seats.

  Straightforward, yes?

  Let me tell you about a different annual fundraising appeal. One that I ran, in fact, during the early years of the Machine Intelligence Research Institute. One difference was that the appeal was conducted over the Internet. And another difference was that the audience was largely drawn from the atheist / libertarian / technophile / science fiction fan / early adopter / programmer / etc. crowd. (To point in the rough direction of an empirical cluster in personspace. If you understood the phrase “empirical cluster in personspace” then you know who I’m talking about.)

  I crafted the fundraising appeal with care. By my nature I’m too proud to ask other people for help; but I’ve gotten over around 60% of that reluctance over the years. The nonprofit needed money and was growing too slowly, so I put some force and poetry into that year’s annual appeal. I sent it out to several mailing lists that covered most of our potential support base.

  And almost immediately, people started posting to the mailing lists about why they weren’t going to donate. Some of them raised basic questions about the nonprofit’s philosophy and mission. Others talked about their brilliant ideas for all the other sources that the nonprofit could get funding from, instead of them. (They didn’t volunteer to contact any of those sources themselves, they just had ideas for how we could do it.)

  Now you might say, “Well, maybe your mission and philosophy did have basic problems—you wouldn’t want to censor that discussion, would you?”

  Hold on to that thought.

  Because people were donating. We started getting donations right away, via Paypal. We even got congratulatory notes saying how the appeal had finally gotten them to start moving. A donation of $111.11 was accompanied by a message saying, “I decided to give a little bit more. One more hundred, one more ten, one more single, one more dime, and one more penny. All may not be for one, but this one is trying to be for all.”

  But none of those donors posted their agreement to the mailing list. Not one.

  So far as any of those donors knew, they were alone. And when they tuned in the next day, they discovered not thanks, but arguments for why they shouldn’t have donated. The criticisms, the justifications for not donating—only those were displayed proudly in the open.

  As though the treasurer had finished his annual appeal, and everyone not making a pledge had proudly stood up to call out justifications for refusing; while those making pledges whispered them quietly, so that no one could hear.

  I know someone with a rationalist cause who goes around plaintively asking, “How come the Raëlian flying-saucer cult can get tens of thousands of members [probably around 40,000] interested in complete nonsense, but we can’t even get a thousand people working on this?”

  The obvious wrong way to finish this thought is to say, “Let’s do what the Raëlians do! Let’s add some nonsense to this meme!” For the benefit of those not immediately stopped by their ethical inhibitions, I will observe that there may be a hundred failed flying saucer cults for every one that becomes famous. And the Dark Side may require non-obvious skills, which you, yes you, do not have: Not everyone can be a Sith Lord. In particular, if you talk about your planned lies on the public Internet, you fail. I’m no master criminal, but even I can tell certain people are not cut out to be crooks.

  So it’s probably not a good idea to cultivate a sense of violated entitlement at the thought that some other group, who you think ought to be inferior to you, has more money and followers. That path leads to—pardon the expression—the Dark Side.

  But it probably does make sense to start asking ourselves some pointed questions, if supposed “rationalists” can’t manage to coordinate as well as a flying saucer cult.

  How do things work on the Dark Side?

  The respected leader speaks, and there comes a chorus of pure agreement: if there are any who harbor inward doubts, they keep them to themselves. So all the individual members of the audience see this atmosphere of pure agreement, and they feel more confident in the ideas presented—even if they, personally, harbored inward doubts, why, everyone else seems to agree with it.

  (“Pluralistic ignorance” is the standard label for this.)

  If anyone is still unpersuaded after that, they leave the group (or in some places, are executed)—and the remainder are more in agreement, and reinforce each other with less interference.

  (I call t
hat “evaporative cooling of groups.”)

  The ideas themselves, not just the leader, generate unbounded enthusiasm and praise. The halo effect is that perceptions of all positive qualities correlate—e.g. telling subjects about the benefits of a food preservative made them judge it as lower-risk, even though the quantities were logically uncorrelated. This can create a positive feedback effect that makes an idea seem better and better and better, especially if criticism is perceived as traitorous or sinful.

  (Which I term the “affective death spiral.”)

  So these are all examples of strong Dark Side forces that can bind groups together.

  And presumably we would not go so far as to dirty our hands with such . . .

  Therefore, as a group, the Light Side will always be divided and weak. Technophiles, nerds, scientists, and even non-fundamentalist religions will never be capable of acting with the fanatic unity that animates radical Islam. Technological advantage can only go so far; your tools can be copied or stolen, and used against you. In the end the Light Side will always lose in any group conflict, and the future inevitably belongs to the Dark.

  I think that a person’s reaction to this prospect says a lot about their attitude towards “rationality.”

  Some “Clash of Civilizations” writers seem to accept that the Enlightenment is destined to lose out in the long run to radical Islam, and sigh, and shake their heads sadly. I suppose they’re trying to signal their cynical sophistication or something.

  For myself, I always thought—call me loony—that a true rationalist ought to be effective in the real world.

  So I have a problem with the idea that the Dark Side, thanks to their pluralistic ignorance and affective death spirals, will always win because they are better coordinated than us.