Reasoning in the absence of definite evidence without going instantaneously completely wrong is really really hard. When you’re learning in school, you can miss one point, and then be taught fifty other points that happen to be correct. When you’re reasoning out new knowledge in the absence of crushingly overwhelming guidance, you can miss one point and wake up in Outer Mongolia fifty steps later.
I am pretty sure that scientists who switch off their brains and relax with some comfortable nonsense as soon as they leave their own specialties do not realize that minds are engines and that there is a causal story behind every trustworthy belief. Nor, I suspect, were they ever told that there is an exact rational probability given a state of evidence, which has no room for whims; even if you can’t calculate the answer, and even if you don’t hear any authoritative command for what to believe.
I doubt that scientists who are asked to pontificate on the future by the media, who sketch amazingly detailed pictures of Life in 2050, were ever taught about the conjunction fallacy. Or how the representativeness heuristic can make more detailed stories seem more plausible, even as each extra detail drags down the probability. The notion of every added detail needing its own support—of not being able to make up big detailed stories that sound just like the detailed stories you were taught in science or history class—is absolutely vital to precise thinking in the absence of definite evidence. But how would a notion like that get into the standard scientific apprenticeship? The cognitive bias was uncovered only a few decades ago, and not popularized until very recently.
Then there’s affective death spirals around notions like “emergence” or “complexity” which are sufficiently vaguely defined that you can say lots of nice things about them. There’s whole academic subfields built around the kind of mistakes that Eliezer18 used to make! (Though I never fell for the “emergence” thing.)
I sometimes say that the goal of science is to amass such an enormous mountain of evidence that not even scientists can ignore it: and that this is the distinguishing feature of a scientist; a non-scientist will ignore it anyway.
If there can exist some amount of evidence so crushing that you finally despair, stop making excuses and just give up—drop the old theory and never mention it again—then this is all it takes to let the ratchet of Science turn forward over time, and raise up a technological civilization. Contrast to religion.
Books by Carl Sagan and Martin Gardner and the other veins of Traditional Rationality are meant to accomplish this difference: to transform someone from a non-scientist into a potential scientist, and guard them from experimentally disproven madness.
What further training does a professional scientist get? Some frequentist stats classes on how to calculate statistical significance. Training in standard techniques that will let them churn out papers within a solidly established paradigm.
If Science demanded more than this from the average scientist, I don’t think it would be possible for Science to get done. We have problems enough from people who sneak in without the drop-dead-basic qualifications.
Nick Tarleton summarized the resulting problem very well—better than I did, in fact: If you come up with a bizarre-seeming hypothesis not yet ruled out by the evidence, and try to test it experimentally, Science doesn’t call you a bad person. Science doesn’t trust its elders to decide which hypotheses “aren’t worth testing.” But this is a carefully lax social standard, and if you try to translate it into a standard of individual epistemic rationality, it lets you believe far too much. Dropping back into the analogy with pragmatic-distrust-based-libertarianism, it’s the difference between “Cigarettes shouldn’t be illegal” and “Go smoke a Marlboro.”
Do you remember ever being warned against that mistake, in so many words? Then why wouldn’t people make exactly that error? How many people will spontaneously go an extra mile and be even stricter with themselves? Some, but not many.
Many scientists will believe all manner of ridiculous things outside the laboratory, so long as they can convince themselves it hasn’t been definitely disproven, or so long as they manage not to ask. Is there some standard lecture that grad students get, of which people see this folly, and ask, “Were they absent from class that day?” No, as far as I can tell.
Maybe if you’re super lucky and get a famous mentor, they’ll tell you rare personal secrets like “Ask yourself which are the important problems in your field, and then work on one of those, instead of falling into something easy and trivial” or “Be more careful than the journal editors demand; look for new ways to guard your expectations from influencing the experiment, even if it’s not standard.”
But I really don’t think there’s a huge secret standard scientific tradition of precision-grade rational reasoning on sparse evidence. Half of all the scientists out there still believe they believe in God! The more difficult skills are not standard!
*
249
No Safe Defense, Not Even Science
I don’t ask my friends about their childhoods—I lack social curiosity—and so I don’t know how much of a trend this really is:
Of the people I know who are reaching upward as rationalists, who volunteer information about their childhoods, there is a surprising tendency to hear things like, “My family joined a cult and I had to break out,” or, “One of my parents was clinically insane and I had to learn to filter out reality from their madness.”
My own experience with growing up in an Orthodox Jewish family seems tame by comparison . . . but it accomplished the same outcome: It broke my core emotional trust in the sanity of the people around me.
Until this core emotional trust is broken, you don’t start growing as a rationalist. I have trouble putting into words why this is so. Maybe any unusual skills you acquire—anything that makes you unusually rational—requires you to zig when other people zag. Maybe that’s just too scary, if the world still seems like a sane place unto you.
Or maybe you don’t bother putting in the hard work to be extra bonus sane, if normality doesn’t scare the hell out of you.
I know that many aspiring rationalists seem to run into roadblocks around things like cryonics or many-worlds. Not that they don’t see the logic; they see the logic and wonder, “Can this really be true, when it seems so obvious now, and yet none of the people around me believe it?”
Yes. Welcome to the Earth where ethanol is made from corn and environmentalists oppose nuclear power. I’m sorry.
(See also: Cultish Countercultishness. If you end up in the frame of mind of nervously seeking reassurance, this is never a good thing—even if it’s because you’re about to believe something that sounds logical but could cause other people to look at you funny.)
People who’ve had their trust broken in the sanity of the people around them seem to be able to evaluate strange ideas on their merits, without feeling nervous about their strangeness. The glue that binds them to their current place has dissolved, and they can walk in some direction, hopefully forward.
Lonely dissent, I called it. True dissent doesn’t feel like going to school wearing black; it feels like going to school wearing a clown suit.
That’s what it takes to be the lone voice who says, “If you really think you know who’s going to win the election, why aren’t you picking up the free money on the Intrade prediction market?” while all the people around you are thinking, “It is good to be an individual and form your own opinions, the shoe commercials told me so.”
Maybe in some other world, some alternate Everett branch with a saner human population, things would be different . . . but in this world, I’ve never seen anyone begin to grow as a rationalist until they make a deep emotional break with the wisdom of their pack.
Maybe in another world, things would be different. And maybe not. I’m not sure that human beings realistically can trust and think at the same time.
Once upon a time, there was something I trusted.
Eliezer18 trusted Science.
&
nbsp; Eliezer18 dutifully acknowledged that the social process of science was flawed. Eliezer18 dutifully acknowledged that academia was slow, and misallocated resources, and played favorites, and mistreated its precious heretics.
That’s the convenient thing about acknowledging flaws in people who failed to live up to your ideal; you don’t have to question the ideal itself.
But who could possibly be foolish enough to question, “The experimental method shall decide which hypothesis wins”?
Part of what fooled Eliezer18 was a general problem he had, an aversion to ideas that resembled things idiots had said. Eliezer18 had seen plenty of people questioning the ideals of Science Itself, and without exception they were all on the Dark Side. People who questioned the ideal of Science were invariably trying to sell you snake oil, or trying to safeguard their favorite form of stupidity from criticism, or trying to disguise their personal resignation as a Deeply Wise acceptance of futility.
If there’d been any other ideal that was a few centuries old, the young Eliezer would have looked at it and said, “I wonder if this is really right, and whether there’s a way to do better.” But not the ideal of Science. Science was the master idea, the idea that let you change ideas. You could question it, but you were meant to question it and then accept it, not actually say, “Wait! This is wrong!”
Thus, when once upon a time I came up with a stupid idea, I thought I was behaving virtuously if I made sure there was a Novel Prediction, and professed that I wished to test my idea experimentally. I thought I had done everything I was obliged to do.
So I thought I was safe—not safe from any particular external threat, but safe on some deeper level, like a child who trusts their parent and has obeyed all the parent’s rules.
I’d long since been broken of trust in the sanity of my family or my teachers at school. And the other children weren’t intelligent enough to compete with the conversations I could have with books. But I trusted the books, you see. I trusted that if I did what Richard Feynman told me to do, I would be safe. I never thought those words aloud, but it was how I felt.
When Eliezer23 realized exactly how stupid the stupid theory had been—and that Traditional Rationality had not saved him from it—and that Science would have been perfectly okay with his wasting ten years testing the stupid idea, so long as afterward he admitted it was wrong . . .
. . . well, I’m not going to say it was a huge emotional convulsion. I don’t really go in for that kind of drama. It simply became obvious that I’d been stupid.
That’s the trust I’m trying to break in you. You are not safe. Ever.
Not even Science can save you. The ideals of Science were born centuries ago, in a time when no one knew anything about probability theory or cognitive biases. Science demands too little of you, it blesses your good intentions too easily, it is not strict enough, it only makes those injunctions that an average scientist can follow, it accepts slowness as a fact of life.
So don’t think that if you only follow the rules of Science, that makes your reasoning defensible.
There is no known procedure you can follow that makes your reasoning defensible.
There is no known set of injunctions which you can satisfy, and know that you will not have been a fool.
There is no known morality-of-reasoning that you can do your best to obey, and know that you are thereby shielded from criticism.
No, not even if you turn to Bayescraft. It’s much harder to use and you’ll never be sure that you’re doing it right.
The discipline of Bayescraft is younger by far than the discipline of Science. You will find no textbooks, no elderly mentors, no histories written of success and failure, no hard-and-fast rules laid down. You will have to study cognitive biases, and probability theory, and evolutionary psychology, and social psychology, and other cognitive sciences, and Artificial Intelligence—and think through for yourself how to apply all this knowledge to the case of correcting yourself, since that isn’t yet in the textbooks.
You don’t know what your own mind is really doing. They find a new cognitive bias every week and you’re never sure if you’ve corrected for it, or overcorrected.
The formal math is impossible to apply. It doesn’t break down as easily as John Q. Unbeliever thinks, but you’re never really sure where the foundations come from. You don’t know why the universe is simple enough to understand, or why any prior works for it. You don’t know what your own priors are, let alone if they’re any good.
One of the problems with Science is that it’s too vague to really scare you. “Ideas should be tested by experiment.” How can you go wrong with that?
On the other hand, if you have some math of probability theory laid out in front of you, and worse, you know you can’t actually use it, then it becomes clear that you are trying to do something difficult, and that you might well be doing it wrong.
So you cannot trust.
And all this that I have said will not be sufficient to break your trust. That won’t happen until you get into your first real disaster from following The Rules, not from breaking them.
Eliezer18 already had the notion that you were allowed to question Science. Why, of course the scientific method was not itself immune to questioning! For are we not all good rationalists? Are we not allowed to question everything?
It was the notion that you could actually in real life follow Science and fail miserably that Eliezer18 didn’t really, emotionally believe was possible.
Oh, of course he said it was possible. Eliezer18 dutifully acknowledged the possibility of error, saying, “I could be wrong, but . . .”
But he didn’t think failure could happen in, you know, real life. You were supposed to look for flaws, not actually find them.
And this emotional difference is a terribly difficult thing to accomplish in words, and I fear there’s no way I can really warn you.
Your trust will not break, until you apply all that you have learned here and from other books, and take it as far as you can go, and find that this too fails you—that you have still been a fool, and no one warned you against it—that all the most important parts were left out of the guidance you received—that some of the most precious ideals you followed steered you in the wrong direction—
—and if you still have something to protect, so that you must keep going, and cannot resign and wisely acknowledge the limitations of rationality—
—then you will be ready to start your journey as a rationalist. To take sole responsibility, to live without any trustworthy defenses, and to forge a higher Art than the one you were once taught.
No one begins to truly search for the Way until their parents have failed them, their gods are dead, and their tools have shattered in their hand.
Post Scriptum: On reviewing a draft of this essay, I discovered a fairly inexcusable flaw in reasoning, which actually affects one of the conclusions drawn. I am leaving it in. Just in case you thought that taking my advice made you safe; or that you were supposed to look for flaws, but not find any.
And of course, if you look too hard for a flaw, and find a flaw that is not a real flaw, and cling to it to reassure yourself of how critical you are, you will only be worse off than before . . .
It is living with uncertainty—knowing on a gut level that there are flaws, they are serious and you have not found them—that is the difficult thing.
*
250
Changing the Definition of Science
New Scientist on changing the definition of science, ungated here:1
Others believe such criticism is based on a misunderstanding. “Some people say that the multiverse concept isn’t falsifiable because it’s unobservable—but that’s a fallacy,” says cosmologist Max Tegmark of the Massachusetts Institute of Technology. He argues that the multiverse is a natural consequence of such eminently falsifiable theories as quantum theory and General Relativity. As such, the multiverse theory stands or fails according to how well these other theories
stand up to observational tests.
[ . . . ]
So if the simplicity of falsification is misleading, what should scientists be doing instead? Howson believes it is time to ditch Popper’s notion of capturing the scientific process using deductive logic. Instead, the focus should be on reflecting what scientists actually do: gathering the weight of evidence for rival theories and assessing their relative plausibility.
Howson is a leading advocate for an alternative view of science based not on simplistic true/false logic, but on the far more subtle concept of degrees of belief. At its heart is a fundamental connection between the subjective concept of belief and the cold, hard mathematics of probability.
I’m a good deal less of a lonely iconoclast than I seem. Maybe it’s just the way I talk.
The points of departure between myself and mainstream let’s-reformulate-Science-as-Bayesianism is that:
(1) I’m not in academia and can censor myself a lot less when it comes to saying “extreme” things that others might well already be thinking.
(2) I think that just teaching probability theory won’t be nearly enough. We’ll have to synthesize lessons from multiple sciences, like cognitive biases and social psychology, forming a new coherent Art of Bayescraft, before we are actually going to do any better in the real world than modern science. Science tolerates errors; Bayescraft does not. Nobel laureate Robert Aumann, who first proved that Bayesians with the same priors cannot agree to disagree, is a believing Orthodox Jew. Probability theory alone won’t do the trick, when it comes to really teaching scientists. This is my primary point of departure, and it is not something I’ve seen suggested elsewhere.
(3) I think it is possible to do better in the real world. In the extreme case, a Bayesian superintelligence could use enormously less sensory information than a human scientist to come to correct conclusions. First time you ever see an apple fall down, you observe the position goes as the square of time, invent calculus, generalize Newton’s Laws . . . and see that Newton’s Laws involve action at a distance, look for alternative explanations with increased locality, invent relativistic covariance around a hypothetical speed limit, and consider that General Relativity might be worth testing.
Rationality- From AI to Zombies Page 109