Expert Political Judgment
Page 1
Expert Political Judgment
Expert Political Judgment
HOW GOOD IS IT? HOW CAN WE KNOW?
New Edition
Philip E. Tetlock
With a new preface by the author
PRINCETON UNIVERSITY PRESS
PRINCETON AND OXFORD
Copyright © 2005 by Princeton University Press
Preface to the new edition © 2017 by Princeton University Press
Published by Princeton University Press, 41 William Street,
Princeton, New Jersey 08540
In the United Kingdom: Princeton University Press, 6 Oxford Street,
Woodstock, Oxfordshire OX20 1TR
All Rights Reserved
First published in 2005
New edition, with a new preface by the author, 2017
Cloth ISBN 978-0-691-17828-8
Paperback ISBN 978-0-691-17597-3
THE LIBRARY OF CONGRESS HAS CATALOGED THE FIRST EDITION OF THIS BOOK AS FOLLOWS
Tetlock, Philip.
Expert political judgment : how good is it? how can we know? / Philip E. Tetlock.
p. cm.
Includes bibliographical references and index.
1. Political psychology. 2. Ideology. I. Title.
JA74.5.T38 2005
320'.01'9—dc22 2004061694
British Library Cataloging-in-Publication Data is available
This book has been composed in Sabon
Printed on acid-free paper. ∞
press.princeton.edu
Printed in the United States of America
10 9 8 7 6 5 4 3 2 1
To Jenny, so alive then;
So alive in our hearts now
Contents
Acknowledgments
ix
Preface
xi
Preface to the 2017 Edition
xvii
CHAPTER 1
Quantifying the Unquantifiable
1
CHAPTER 2
The Ego-deflating Challenge of Radical Skepticism
25
CHAPTER 3
Knowing the Limits of One’s Knowledge: Foxes Have Better Calibration and Discrimination Scores than Hedgehogs
67
CHAPTER 4
Honoring Reputational Bets: Foxes Are Better Bayesians than Hedgehogs
121
CHAPTER 5
Contemplating Counterfactuals: Foxes Are More Willing than Hedgehogs to Entertain Self-subversive Scenarios
144
CHAPTER 6
The Hedgehogs Strike Back
164
CHAPTER 7
Are We Open-minded Enough to Acknowledge the Limits of Open-mindedness?
189
CHAPTER 8
Exploring the Limits on Objectivity and Accountability
216
Methodological Appendix
239
Technical Appendix
Phillip Rescober and Philip E. Tetlock
273
Index
313
Acknowledgments
THE JURY is out on just how much bad judgment I showed by undertaking the good-judgment project. The project dates back to the year I gained tenure and lost my generic excuse for postponing projects that I knew were worth doing, worthier than anything I was doing back then, but also knew would take a long time to come to fruition. As I write twenty years later, the data are still trickling in and the project now threatens to outlast not just my career but me. Some long-term forecasts that experts offered will not come due until 2026. But most of the data are tabulated, some surprising patterns have emerged, and I see no reason for delaying the write-up into my retirement.
Of course, a project of this duration requires the cooperation of many people over many years. My greatest collective debt is to the thoughtful professionals who patiently worked through the often tedious batteries of questions on what could have been, what is, and what might yet be. I told them at the outset that I did not intend to write a book that named names, or that, by exploiting hindsight bias, incited readers to glorify those who got it right or ridicule those who got it wrong. I promised strict confidentiality. The book that would emerge from this effort would be variable-centered, not person-centered. The focus would be on the links between how people think and what they get right or wrong, at various junctures, in a kaleidoscopically shifting world. I realize that the resulting cognitive portrait of expert political judgment is not altogether flattering, but I hope that research participants, even the “hedgehogs” among them, do not feel shabbily treated. I level no charges of judgmental flaws that do not also apply to me.
Another great debt is to the many colleagues who offered methodological and theoretical advice that saved me from making an even bigger fool of myself than I may have already done. Barbara Mellers, Paul Tetlock, and Phillip Rescober offered invaluable guidance on how to design measures of forecasting skill that were sensitive to the variety of ingenious objections that forecasters raised when either high probability events failed to materialize or low probability events did materialize. And colleagues from several disciplines—including psychology, political science, economics, history, and the hybrid field of intelligence analysis—made suggestions at various junctures in this long journey that, in my opinion at least, improved the final product. I cannot remember the source of every insightful observation at every stage of this project, but this list should include in roughly chronological order from 1984 to 2004: Peter Suedfeld, Aaron Wildavsky, Alexander George, George Breslauer, Danny Kahneman, Robyn Dawes, Terry Busch, Yuen Foong Khong, John Mercer, Lynn Eden, Amos Tversky, Ward Edwards, Ron Howard, Arie Kruglanski, James March, Joel Mokyr, Richard Herrmann, Geoffrey Parker, Gary Klein, Steve Rieber, Yaacov Vertzberger, Jim Goldgeier, Erika Henik, Rose McDermott, Peter Scoblic, Cass Sunstein, and Hal Arkes. In the final phases of this project, Paul Sniderman and Bob Jervis played a particularly critical role in helping to sharpen the central arguments of the book. Needless to say, though, none of the aforementioned bears responsibility for those errors of fact or interpretation that have persisted despite their perceptive advice.
I also owe many thanks to the many former and current students who have worked, in one capacity or another, on various components of this project. They include Charles McGuire, Kristen Hannum, Karl Dake, Jane Bernzweig, Richard Boettger, Dan Newman, Randall Peterson, Penny Visser, Orie Kristel, Beth Elson, Aaron Belkin, Megan Berkowitz, Sara Hohenbrink, Jeannette Porubin, Meaghan Quinn, Patrick Quinn, Brooke Curtiss, Rachel Szteiter, Elaine Willey, and Jason Mills. I also greatly appreciate the staff support of Deborah Houy and Carol Chapman.
Turning to institutional sponsors, this project would have been impossible but for generous financial and administrative support from the following: the John D. and Catherine T. MacArthur Foundation Program in International Security, the Open Philanthropy Project, the Institute of Personality and Social Research of the University of California, Berkeley, the Institute on Global Conflict and Cooperation at the University of California, the Center for Advanced Study in the Behavioral Sciences in Palo Alto, the Mershon Center of the Ohio State University, the Social Science Research Council, the National Science Foundation, the United States Institute of Peace, the Burtt Endowed Chair in the Psychology Department at the Ohio State University, and the Mitchell Endowed Chair at the Haas School of Business at the University of California, Berkeley.
Finally, I thank my family—especially, Barb, Jenny, and Paul—for their infinite forbearance with my workaholic ways.
Preface
AUTOBIOGRAPHICAL exercises that explore why the researcher opted to go forward with one project rather than another have often struck m
e as self-dramatizing. What matters is the evidence, not why one collected it. Up to now, therefore, I have hewed to the just-the-facts conventions of my profession: state your puzzle, your methods, and your answers, and exit the stage.
I could follow that formula again. I have long been puzzled by why so many political disagreements—be they on national security or trade or welfare policy—are so intractable. I have long been annoyed by how rarely partisans admit error even in the face of massive evidence that things did not work out as they once confidently declared. And I have long wondered what we might learn if we approached these disputes in a more aggressively scientific spirit—if, instead of passively watching warring partisans score their own performance and duly pronounce themselves victorious, we presumed to take on the role of epistemological referees: soliciting testable predictions, scoring accuracy ourselves, and checking whether partisans change their minds when they get it wrong.
I initially implemented my research plan tentatively, in a trial-and-error fashion in small-scale forecasting exercises on the Soviet Union in the mid-1980s, and then gradually more boldly, in larger-scale exercises around the world over the next decade. My instinct was to adopt and, when necessary, adapt methods of keeping score from my home discipline of psychology: correspondence measures of how close political observers come to making accurate predictions and logical-process measures of the degree to which observers play fair with evidence and live up to reputational bets that require them to update their beliefs.
Without giving too much away, I can say that surprises are in store. We shall discover that the best forecasters and timeliest belief updaters shared a self-deprecating style of thinking that spared them some of the big mistakes to which their more ideologically exuberant colleagues were prone. There is often a curiously inverse relationship between how well forecasters thought they were doing and how well they did.
I could now exit the stage. But the project makes more sense when traced to its origins: my first close-up contact with the ingenuity and determination that political elites display in rendering their positions impregnable to evidence. The natural starting point is a 1984 meeting at the National Research Council, the administrative branch of the National Academy of Sciences. I was a freshly tenured professor from Berkeley and the most junior (nonmember of the academy) member of the committee. The committee had been convened—as academic committees often are—to midwife the birth of another committee. This new committee would have an ambitious—critics said pretentious—mandate: to explore the contributions of the social sciences, not to the humdrum, usual-suspect problems of early childhood education or affirmative action, but to rescuing civilization itself from nuclear incineration.
Just because we want an answer, even desperately want one, does not mean we have an answerable question. Science is, by one famous definition, the art of the solvable,1 and I was not alone in fearing that our humanitarian reach exceeded our scientific grasp. Although the clock on the cover of the Bulletin of the Atomic Scientists had edged closer to midnight than at any other time aside from the Cuban missile crisis of October 1962, it was obvious to most around the table that we had no metric for gauging our proximity to nuclear war, something that had not yet happened and might never happen. The clock-setters were guessing. Indeed, it was not clear how the classic methods of clarifying causality, experimental and statistical control, could even be applied to explain the nonoccurrence of an event (nuclear war) that qualified as sui generis if ever one did, or to the challenge of extrapolating its continued nonoccurrence.2
Thoughtful activists replied that too much thought can be paralyzing. One speaker dismissed “worrywarts” who find themselves hurtling down a mountain highway in a bus with a demented driver at the wheel, but who would rather debate the sines and cosines on the next curve than wrestle control from the maniac (an allusion to Ronald Reagan) who was about to send them plunging to their doom.3 Another speaker posed two possible futures: in one you must explain to your irradiated and slowly dying children why you did nothing as the world edged toward nuclear apocalypse, and in the other you must explain to your bemused colleagues why you thought the end was imminent when everything worked out well. Easy choice, he thought.
Academic administrators defuse debates of this sort by artful obfuscation. The activists got a committee but not the putschist committee they wanted—one that would issue thundering, Linus Pauling–style indictments of warmongers. The worrywarts, who doubted the wisdom of any committee, got the kind of committee they would have wanted had they wanted one: a politically innocuous but academically respectable forum for assessing the sines and cosines on the curvy political road ahead.
I took it upon myself to canvas professional opinion on the American-Soviet relationship. What did the experts know, or think they know, about Soviet intentions? How did they judge whether American policy had struck the right balance between deterrence and reassurance in dealing with the Soviets? Looking back over the previous forty years, did they see any, or perhaps many, “missed opportunities” to promote peaceful cooperation? In probing these “foundational” beliefs, I was struck by how frequently influential observers offered confident, but flatly contradictory, assessments that were impervious to the arguments advanced by the other side. Hawks saw the Soviet Union as an evil empire that had to be contained through deterrence; doves saw a series of misunderstandings rooted in rigid mind-sets and exploited by self-serving interest groups; and self-styled owls flit between these polar oppositions, crafting proposals that blended hawks’ determination to deter and doves’ desire to reassure.4
Two decades later, it is hard to re-create the mood of apprehension among doves. They felt that the Reagan administration was dragging us precariously close to the precipice.5 In hindsight, it is tempting to be condescending. We now know that in March 1985, Mikhail Gorbachev became General Secretary of the Communist Party and introduced sweeping reforms. The fear of all-out nuclear war dissipated as thoughtful observers turned their attention to nuclear terrorism and about what would happen to the massive Soviet stock of weapons of mass destruction when the Russian government could no longer pay the military custodians of those weapons their pittance salaries. The epicenter of controversy shifted.6 Animated debates between hawks and doves gave way to spirited exchanges over managing transitions from socialism to capitalism, coping with resurgent nationalism and fundamentalism, and saving humanity from ecocatastrophes. Cold war thinking was passé.
Was it foolish to have gotten so worked up? Many conservatives believe so: the National Research Council erred in creating a committee that lent even limited legitimacy to a gaggle of academic Chicken Littles. One respected colleague chided me: “So the sky was not falling.” The much ridiculed Reagan administration was right to have kept on upping the ante in the geopolitical poker game until the Kremlin folded out.7
Liberals saw things differently.8 They worried that bellicose rhetoric and a massive defense buildup were setting in motion a cycle of hostility in which each side exaggerates the aggressive intent of the other and, by preparing for the worst, guarantees the worst. Many, moreover, have not changed their minds: they still insist that the cold war would have ended just as swiftly in a world with a two-term Carter presidency and Mondale follow-ups. Conservative “triumphalism” reminded one prominent scientist of a man who wins a round of Russian roulette and proclaims himself a genius. We were lucky that Gorbachev, rather than a vodka-guzzling neo-Stalinist, was waiting in the wings to take over. This scholar conjured up scenarios in which the Politburo, angered by American provocations, elevated an “apparatchik thug” to general secretary in 1985 and began to play major-league forms of the game of nuclear brinkmanship that North Korea, in minor-league form, is playing today. Civilization would soon be tottering on the brink.
Here we see the first, but not the last, example of a popular belief system defense among inaccurate forecasters, the close-call counterfactual: “Well, I predicted x and x did not happen, but it almo
st did. You who laugh reveal the poverty of your historical imaginations.”9 Declining empires do not usually accept relegation to the dustbin of history as peacefully as the Soviet Union. Talk about the inevitability of collapse, about how internal weakness and external pressure forced the hands of the Soviet leadership, tells us less about the probability distribution of possible worlds than it does about the self-deceptive tricks that hindsight plays on the mind. We too easily convince ourselves that we knew all along what was going to happen when, in fact, we were clueless.
Political partisans will sometimes suspect me of playing favorites. In some quarters, I am already under a cloud of suspicion. Why dwell on a forecasting fiasco for the left? Do I have a neoconservative agenda of showing that, not only were the Cassandras of the left wrong, they try to cover up this massive mistake with gibberish about counterfactual worlds in which events dovetail suspiciously smoothly with the apocalyptic movie-script scenarios that their Hollywood friends penned twenty years ago?10
But, alas for the political right, the turning point in Soviet political history in March 1985, provides a less than ideal opportunity for gloating. The data reveal ample grounds for embarrassment across the spectrum of once-fashionable opinion. Conservatives were no better forecasters than liberals. In fact, many were slower than liberals to acknowledge Gorbachev’s commitment to reform, some dismissing him as an apparatchik in “Gucci garb” right up to the coup attempt of August 1991. They were blindsided by the emergence of a mutant reformer like Gorbachev from deep within the bowels of a totalitarian system that they had postulated to be drearily uniform and infallibly self-reproducing. Contrary to Richard Pipes and Jeanne Kirkpatrick, the Soviets had not mastered the technology of ideological cloning.11 The pudgy grey gerontocrats of the Politburo, dutifully lined up in their furry hats on the Kremlin wall every November 7, looked alike but did not think alike. To justify their tardiness on the “Gorbachev call,” conservatives often resorted to another popular belief system defense: “OK, I made a mistake, but it was the right mistake.” The error of underestimating the Soviet threat was more serious than that of overestimating it.12