Expert Political Judgment

by Philip E. Tetlock

  Role reversal exercises came more naturally to foxes who recognized that what looks rational to us might look foolish or unfair to them, and that what looks irrational to us might seem honorable or necessary to them. Foxes tended to see the world as a shifting mixture of self-fulfilling and self-negating prophecies: self-fulfilling ones in which success breeds success and failure, failure but only up to a point, and then self-negating prophecies kick in as people recognize that things have gone too far. Foxes saw wisdom in the old adage that we never know we have had enough until we have had more than enough, and they agreed with skeptics that it is impossible to predict the moment of epiphany—the “magic moment” when, on an evening stroll, a Gorbachev will turn to a Shevardnadze and say, “We can’t go on living like this anymore.”

  China (1992). Optimists thought that China could sustain annual growth rates of 10 percent indefinitely. But they split on the implications of that growth. Some foresaw an easing of repression and emergence of democracy. Others thought growth would prop up hardliners and subsidize the military apparatus that keeps the Communists in power. The preponderance of sentiment, however, favored the former view: economic pluralism and rising living standards would lead to political pluralism just as it had in South Korea, Taiwan, and Singapore. The burgeoning middle class would in due course topple the tyrants who brought them prosperity.

  Hedgehog optimists dwelled on the parallels between China and the miracle economies of the “Asian Tigers,” parallels they justified by pointing to cultural and political commonalities: a Confucianist legacy of respect for education, thrift, and duty and an authoritarian legacy of single-party rule.

  Hedgehog pessimists replied that big improvements were “easier” earlier because the baseline of comparison was so low and inefficiency so blatant. The pessimists focused on rampant corruption, on the risk of chaos in future power struggles and on the lack of political legitimacy. Some even thought China might fall apart. Invoking the specters of the British, French, and Soviet empires, one characterized China as the “last of the great multiethnic empires” encompassing “Buddhist Tibet, Muslim Xinjiang, Korean cities in Manchuria, Cantonese Guangdong, and cosmopolitan Hong Kong.” China’s early twenty-first-century future would resemble its early twentieth-century past: “a patchwork of fiefdoms under rival warlords.” Another observer compared Chinese Communism to a “decrepit mansion held together by shoddy repair jobs…. [Its] leaders are scrambling to keep up with deferred maintenance. But they will eventually have to justify their existence. When the economy stalls, and it must, the government will resort to the last refuge of scoundrels [patriotism]…. China has boundary disputes not just with Taiwan but also with Vietnam, Russia, and India. In the next decade, the Asian equivalent of NATO will arise with the purpose of containing China.”

  The foxes warned against overreacting. One noted in fortune-cookie fashion: “Things are rarely as bad as they look in the troughs or as good as they look at the peaks. There was despair after the Tiananmen massacre. There will be more causes for despair.” Foxes also split over how wisely the Chinese leadership would cope with coming crises. The dominant view was that Deng Xiaoping was “awesomely shrewd” (“he had thirty IQ points on Mao”) and had picked successors who shared his game plan. The dominant prediction was therefore “repressive stability,” “robust growth,” and occasionally tense but mostly businesslike relations with the United States. Foxes reserved the right to change their minds if reactionaries seized control, reversed movement toward free markets, and picked fights that galvanized an alliance against China. The foxes also agreed with pessimists that, as the income gap grows between rural and urban dwellers, a vast migratory labor class from the countryside will start demanding jobs in the cities, producing a surge in crime and unrest (“Mao’s revenge”). On its path to world-power status, China might pass “through several Tiananmen-magnitude crises. It will be a bumpy ride but China can stay on the trajectory toward superpower status if the post-Deng leadership keeps an even policy keel.”44 This fox paraphrased former U.S. Treasury secretary Larry Summers: “When the history of the twentieth century is written one hundred years from now, the most significant event will be the revolutionary changes in China…. For more than a century, the United States has been the world’s largest economy. The only nation with a chance of surpassing it in the next generation in absolute scale is China.” Indeed, if China were to hit Taiwan’s per capita income, its economy would be larger than all industrialized countries in the world combined. It would be “like the rise of Japan, except China has nuclear weapons and ten times the population.”

  A fox, sympathetic to hegemonic transition theory, gets the last word: “If Deng’s successors play their cards right, we are heading into a Sinocentric world by the mid-twenty-first century, at least a world in which China is America’s principal rival. Managing such power transitions is a delicate task that many statesmen have failed, leaving wars in their wake. The Beijing leadership could set back the clock for the rise of China by decades, even centuries, if they throw around their weight carelessly…. My guess is that they will be too smart to pass up this historical opportunity by picking unnecessary fights.” Here is the trademark eclecticism of foxes. The speaker applied a macrotheory (hegemonic transition) in a way that allowed for microvariables (leadership decisions) to gate us into alternative futures. The conversation closed with another fox trademark: the hedge. “Of course, there are elements of chance…. Deng’s successors might revert through regression toward the mean to the average intelligence of politicians.”

  CLOSING OBSERVATIONS

  Quantitative and qualitative methods converge on a common conclusion: foxes have better judgment than hedgehogs. Better judgment does not mean great judgment. Foxes are not awe-inspiring forecasters: most of them should be happy to tie simple extrapolation models, and none of them can hold a candle to formal statistical models. But foxes do avoid many of the big mistakes that drive down the probability scores of hedgehogs to approximate parity with dart-throwing chimps. And this accomplishment is rooted in foxes’ more balanced style of thinking about the world—a style of thought that elevates no thought above criticism.

  By contrast, hedgehogs dig themselves into intellectual holes. The deeper they dig, the harder it gets to climb out and see what is happening outside, and the more tempting it becomes to keep on doing what they know how to do: continue their metaphorical digging by uncovering new reasons why their initial inclination, usually too optimistic or pessimistic, was right. Hedgehogs are thus at continual risk of becoming prisoners of their preconceptions, trapped in self-reinforcing cycles in which their initial ideological disposition stimulates thoughts that further justify that inclination which, in turn, stimulates further supportive thoughts.45

  There are intriguing parallels between the evidence on how foxes outperformed hedgehogs and the broader literature on how to improve forecasting. We learn from the latter that (a) the average predictions of forecasters are generally more accurate than the majority of forecasters from whom the averages were computed; (b) trimming outliers (extremists) further enhances accuracy; (c) one can do better still by using the Delphi technique for integrating experts’ judgments in which one persuades experts to advance anonymous predictions and arguments for those predictions, one then circulates everyone’s predictions and arguments to everyone else (so everyone has a chance to reflect but no one has a chance to bully), and one continues the process until convinced the process has reached the point of diminishing returns.46 These results dovetail with the cognitive interpretation of the fox-hedgehog performance gaps: foxes do better because they are moderates who factor conflicting considerations—in a flexible, weighted-averaging fashion—into their final judgments.47

  Overall, chapter 3 makes a strong case that the foxes’ “victory” was a genuine achievement. We looked for good judgment and found it—mostly among the foxes. And, interestingly, this does not appear to be where most of the media are looking. Hedge
hog opinion was in greater demand from the media, and this was probably for the reason noted in chapter 2: simple, decisive statements are easier to package in sound bites. The same style of reasoning that impairs experts’ performance on scientific indicators of good judgment boosts experts’ attractiveness to the mass market–driven media.

  It is premature, though, to segue into social commentary. Not everyone is ready to concede that foxes do better because they are better cognitively equipped for making sense of the world. One pocket of resistance is concentrated among psychologists who subscribe to the argument that fast-and-frugal heuristics—simple rules of thumb—perform as well as, or better than, more complex, effort-demanding algorithms.48 Another pocket of resistance is concentrated among policy makers who prefer one-handed advisers—and among political scientists and historians who defend that preference.49 But, whatever the roots of the resistance, the resisters—if they are to engage the scientific debate—need to identify logical or empirical flaws in the arguments advanced here—flaws sufficiently severe that they justify dismissing the consistently large performance gaps between hedgehogs and foxes as illusory.

  Some pro-hedgehog reviewers of this manuscript have attempted to identify such flaws—and compelled me to sharpen my own case (our most relentless critics sometimes teach us the most).50 One critique maintains that we will discover that hedgehogs are every bit as discerning observers as foxes when we factor in the different error-avoidance priorities of the two groups (hence the need for value-adjusted probability scores). Another critique complains about an uneven playing field: hedgehog experts “lost” because they specialized in more unpredictable regions of the world and were dealt tougher assignments (hence the need for difficulty-adjusted forecasting scores). Yet a third critique calls for giving more weight to the defenses that forecasters offered when the unexpected occurred—credit for being almost right (hence the need for fuzzy-set adjustments). Fairness requires giving all appeals a hearing, but we defer the hearing until chapter 6, where defenders of hedgehogs have the chance to rebut not just the evidence in this chapter, but also that in chapters 4 and 5.

  1 Factor analysis selects factors by initially maximizing the likelihood that the variance-covariance matrix is generated by a single factor plus normally distributed disturbances. (Note 1 con’d from p. 69) Extracting the single factor from the data yields a residual variance-covariance matrix from which the additional factors are selected by repeating the likelihood maximization process. After applying several factoring methods and rotation procedures, we settled on a maximum likelihood solution with quartimin rotation (that did not force factors into orthogonal alignments). The retention of three factors was justified by the pattern of decline in eigenvalues and by the improving fit to the data from more formal goodness-of-fit tests. For details, see L. R. Fabrigar, D. T. Wegener, R. C. MacCallum, and E. J. Strahan, “Evaluating the Use of Exploratory Factor Analysis in Psychological Research,” Psychological Methods 4(3) (1999): 272–99. F. J. Floyd and K. F. Widaman, “Factor Analysis in the Development and Refinement of Clinical Assessment Instructions,” Psychological Assessment 7(3) (1995): 286–99.

  2 D. P. Moynihan, Pandemonium (New York: Oxford University Press, 1997).

  3 The first and third factors were correlated (r = .43). Boomsters tilted to the right, doomsters to the left. The second factor was moderately correlated (r = 0.27) with the first but negligibly with the third (.09). Realists and materialists favored the right; institutionalists and idealists, the left.

  4 Regression analyses, technically more appropriate, tell the same story. The tertile (and later quartile) splits do, though, simplify presentation.

  5 Berlin, “The Hedgehog and the Fox.”

  6 Of course, impressive though the correspondence is between our empirical dimension and Berlin’s conceptual distinction, the mapping is imperfect. Factor analysis transforms Sir Isaiah’s dichotomy into a measurement continuum that treats “hedgehogness” and “foxiness” as matters of degree, not all or none. Psychologists are familiar with this dimension of human personality. It bears a family resemblance to the generic openness factor in multivariate studies of personality structure as well as to several measures of cognitive style in the psychological literature, especially those of need for closure and integrative complexity (see, for example, O. P. John, “The ‘Big Five’ Factor Taxonomy: Dimensions of Personality in the Natural Language and in Questionnaires,” in Handbook of Personality, ed. L. A. Pervin (New York: Guilford Press, 1990); A. W. Kruglanski and D. M. Webster, “Motivated Closing of the Mind: ‘Seizing’ and ‘freezing,’ ” Psychological Review 103 (1996): 263–68; also P. Suedfeld and P. E. Tetlock, “Cognitive Styles,” in Blackwell International Handbook of Social Psychology, vol. 1, ed. A. Tesser and N. Schwartz (London: Blackwell, 2001). High need-for-closure, integratively simple individuals are like Berlin’s hedgehogs: they dislike ambiguity and dissonance in their personal and professional lives, place a premium on parsimony, and prefer speedy resolutions of uncertainty that keep prior opinions intact. Low need-for-closure, integratively complex individuals are like Berlin’s foxes: they are tolerant of ambiguity and dissonance, curious about other points of view, and open to the possibility they are wrong.

  7 Political psychologists have a long-standing interest in the linkages between cognitive style and political ideology. The current results run against the dominant grain in this literature. Most studies have found that those on the political right tend to score higher on psychological measures of preferences for simplicity and closure than those on the left and the center of the ideological spectrum. (See J. T. Jost, J. Glaser, A Kruglanski, and F. Sulloway, “Political Conservatism as Motivated Social Cognition,” Psychological Bulletin 129 [2003]: 339–75.) By contrast, we have found that those on the left and right ends of the spectrum in our sample obtain roughly comparable scores on such measures and both groups score higher than those toward the center of the spectrum. It would be a mistake though to make too much of this inconsistency—a mistake for at least three reasons. First, the research literature is not monolithic. In my own past work, I have found support for both the rigidity-of-the-right and the ideologue hypotheses. Much hinges on the proportions of true believers from the left and the right in one’s sample. (See P. E. Tetlock, “Cognitive Structural Analysis of Political Rhetoric: Methodological and Theoretical Issues, in Political Psychology: A Reader, ed. S. Iyengar and W. J. McGuire, 380–407 (Durham, NC: Duke University Press, 1992.) Second, the research literature offers no precise guidance on how far to the left or right one must go to observe hypothesized shifts in cognitive style. Third, causality is murky. On the one hand, cognitive style may shape the content of one’s political views. Hedgehogs may be drawn to all-encompassing abstractions and foxes may be drawn to blurrier compromise positions. On the other hand, the content of one’s political views may shape one’s style of reasoning. Cognitive style may simply be a by-product of the moral-political values that we hold dear and the frequency with which the world forces us to make tough choices. Reciprocal determinism is probably at work.

  8 As described in the Technical Appendix, one way to estimate proportion of variance predicted (an omniscience ratio) is to divide DI scores by the total variability in outcomes (VI) and multiplying by 100.

  9 The worst-performing professionals lose overall to the chimp because they win by too small a margin on discrimination to compensate for the size of their defeat on calibration. But they still dominate the briefly briefed undergraduates—a sign that, although we reach the point of diminishing predictive returns for knowledge quickly, well-informed hedgehog ideologues derive some predictive benefits from their impressive stores of knowledge.

  10 As noted in chapter 2 and the Technical Appendix, outcomes varied in the degree to which they could be predicted from knowledge of their own recent past or the recent past of other lagged variables in the dataset (squared multiple correlations from time series models ranging from .21 to .78). The generali
zability of the fox advantage across outcomes casts doubt on the arguments that (a) foxes were merely more adept at picking the lowest-hanging predictive fruit (an argument that, even if true, hardly casts hedgehogs in a flattering light); (b) foxes “lucked out” and, because they were closer to being right on a few outcomes that were extensively connected to other outcomes, they enjoyed the benefits of cascading. It is worth emphasizing that our statistical tests avoid capitalizing on cascading by averaging across large numbers of observations and making highly conservative assumptions about degrees of freedom.

  11 The similar profile of correlates for calibration and discrimination should be no surprise given the substantial correlation, –.6, between the two indicators.

  12 These statistical tests are based on mixed-design, repeated-measures analyses of variance that took the form of 4 (quartile split on cognitive-style scale) × 2 (expert versus dilettante) × 2 (moderate versus extremist) × (short-range versus long-range predictions) designs that allow for correlations between the repeated-measures variables. The tests rest on conservative assumptions about degrees of freedom (each short-term or long-term calibration or discrimination score is itself an average derived from, on average, thirty forecasts across two states).