Expert Political Judgment

by Philip E. Tetlock

  13 Kruglanski and Webster, “Motivated Closing of the Mind.”

  14 C-y Chiu, M. W. Morris, Y-y Hong, and T. Menon, “Motivated Cultural Cognition: The Impact of Implicit Cultural Theories on Dispositional Attribution Varies as a Function of Need for Closure,” Journal of Personality and Social Psychology 78(2) (2000): 247–59; P. E. Tetlock, “Close-call Counterfactuals and Belief System Defenses: I Was Not Almost Wrong but I Was Almost Right,” Journal of Personality and Social Psychology 75 (1998): 639–52.

  15 Suedfeld and Tetlock, “Cognitive Styles,” in Tesser and Schwartz, Blackwell International Handbook of Social Psychology. To prevent my own biases from contaminating assessments of thinking styles, three additional coders—unaware of the hypotheses being tested and the sources of the material—applied the same coding rules to the texts. Depending on which variables and which subsamples of text were examined, interjudge reliability ranged between .76 and .89. Disagreements were averaged out for data analysis purposes.

  16 Readers unmoved by merely correlational evidence may be more persuaded by experimental evidence that forecasters forced to think in more complex ways (and to use multiple historical analogies) made more accurate predictions. See K. C. Green and J. S. Armstrong, “Structured Analogies for Forecasting,” (Monash University Econometrics Working Paper 17/04, 2004), full text available at www.conflictforecasting.com.

  17 Berlin, “The Hedgehog and the Fox.”

  18 C. Hempel, “The Function of General Laws in History,” in The Philosophy of History, ed. Patrick Gardiner (New York: Oxford University Press, 1942).

  19 For conflicting postmortems on Yugoslavia, see “Ex-Yugoslavs on Yugoslavia: As They See It,” Economist 338 (1996): 5–6; M. Glenny, The Balkans: Nationalism, War, and the Great Powers, 1804–1999 (New York: Viking, 2000); J. Gowa, Triumph of the Lack of Will: International Diplomacy and the Yugoslav War (New York: Columbia University Press, 1997); D. Rieff, Slaughterhouse: Bosnia and the Failure of the West (New York: Simon & Schuster, 1995); P. Akhavaran and R. Howse, eds., Yugoslavia, the Former and Future: Reflections by Scholars from the Region (Washington: Brookings Institute, 1993); L. J. Cohen, Broken Bonds: Yugoslavia’s Disintegration and Balkan Politics in Transition (Boulder, CO: Westview Press, 1995); L. J. Cohen, The Serpent in the Bosom: The Rise and Fall of Slobodan Milosovic (Boulder, CO: Westview Press, 2000).

  20 For conflicting views of the viability of the European Monetary Union, see Timothy G. Ash, “The European Orchestra,” New York Review of Books, May 17, 2001, 60–65; P. Gowan and P. Anderson, eds., The Question of Europe (Verso, London, 1997); M. Feldstein, “Europe Can’t Handle the Euro,” Wall Street Journal, February 8, 2000; “A Survey of Europe: A Work in Progress,” Economist, October 23, 1999; P. De Grauwe, Economics of Monetary Union, 4th ed. (Oxford: Oxford University Press, 2000); S. F. Overturk, Money and European Union (New York: St. Martin’s Press, 1997); K. Dyson, ed., The Road to Maastricht: Negotiating Economic and Monetary Union (Oxford: Oxford University Press, 1999); B. Eichengreen, The Political Economy of European Monetary Integration (Boulder, CO: Westview Press, 2000); J. Story and I. Walter, eds., Political Economy of Financial Integration in Europe: The Battle of the Systems (Cambridge: MIT Press, 1998); K. R. McNamara, The Currency of Ideas: Monetary Politics in the European Union (Ithaca, NY: Cornell University Press, 1998); J. Mills, Europe’s Economic Dilemma (New York: Macmillan, 1998).

  21 For conflicting views of what went wrong in the first post-Communist decade in Russia, see R. L. Garthoff, “The United States and the New Russia: The First Five Years,” Current History 96(612) (1997): 305–12; M. I. Goldman, “Russia’s Reform Effort: Is There Growth at the End of the Tunnel?” Current History 96(612) (1997): 313–18; M. McFaul, “Democracy Unfolds in Russia,” Current History 96(612) (1997): 319–25; G. Soros, “Who Lost Russia?” New York Review, April 2000, 10–16; T. E. Graham Jr., “The Politics of Power in Russia,” Current History 98(630) (1999): 316–21; J. R. Millar, “The De-development of Russia,” Current History 98(630) (1999): 322–27; R. P. Powaski, “Russia: The Nuclear Menace Within,” Current History 98(630) (1999): 340–45; “Russia: Things Fall Apart; the Centre Cannot Hold,” The World in 1999 Economist (2000): 60; “The Battle of Russia’s Capitalisms,” Economist 344 (1997): 14; Archie Brown, Contemporary Russian Politics: A Reader (Oxford: Oxford University Press, 2001); David D. Laitin, Identity in Formation: The Russia-speaking Populations in the Near Abroad (Ithaca, NY: Cornell University Press, 1998).

  22 Some influential historians argue that those who invoke the Weimar analogy fail to appreciate how unlikely the Hitler outcome was even toward the very end of the Weimar Republic. (H. Turner, Hitler’s 30 Days to Power [New Haven, CT: Yale University Press, 1997].)

  23 For contending views of the future of Kazakhstan, see R. C. Kelly, Country Review, Kazakhstan 1998/1999 (New York: Commercial Data International, 1998); M. Alexandrov, Uneasy Alliance: Relations between Russia and Kazakhstan in the Post-Soviet Era, 1992–1997 (Westport, CT: Greenwood Press, 1999); E. Gurgen, H. Sniek, J. Craig, and J. McHugh, eds., “Economic Reforms in Kazakhstan, Kyrgyz Republic, Tajikstan, Turkmenistan, and Uzbekistan” (Occasional Paper, International Monetary Fund No. 183).

  24 For competing perspectives on how Poland and other East European economies should have managed the post-Communist transition to market economies, see H. Kierzkowski, M. Okolski, and W. Stanislaw, eds., Stabilization and Structural Adjustment in Poland (London: Routledge, 1993); F. Millard, Politics and Society in Poland (London: Routledge, 1999); K. Cordell, Poland and the European Union (London: Routledge, 2000); R. F. Starr, ed., Transition to Democracy in Poland, 2nd ed. (New York: St. Martin’s Press, 1998); J. Adam, Social Costs of Transformation to a Market Economy in Post-Socialist Countries (New York: Palgrave, 2000).

  25 A RAND report of the period captures the range of possible futures deemed plausible by our specialists: Jonathan Pollack and Chung Min Lee, Preparing for Korean Unification: Scenarios and Implications (Santa Monica: RAND, 1999). For regional ramifications, see H. Chang, “South Korea: Anatomy of a Crisis,” Current History 97(623) (1998): 437–41; T. Inoguchi and G. B. Stillman, eds., North-East Asian Regional Security (Tokyo: United Nations University Press, 1997); D. Levin, “What If North Korea Survives?” Survival 39 (1997–98): 156–74.

  26 For premature announcements of Castro’s demise as well as analyses of how Castro outlasted his Soviet patrons, see A. Oppenheimer, Castro’s Final Hour: The Secret Story Behind the Coming Downfall of Communist Cuba (New York: Simon & Schuster, 1993); D. J. Fernandez, Cuba and the Politics of Passion (Austin: University of Texas Press, 2000); E. A. Cardoso, Cuba After Communism (Cambridge: MIT Press, 1992); H. M. Erisman and J. M. Kirk, Cuba’s Foreign Relations in a Post-Soviet World (Gainesville: University Press of Florida, 2000); D. E. Schulz, ed., Cuba and the Future (Westport, CT: Greenwood, 1994).

  27 For samplings from this rancorous debate: J. K. Aburish, The Rise, Corruption and Coming Fall of the House of Saud (St. Martin’s Press, 1996); H. Khashan, Arabs and the Crossroads: Political Identity and Nationalism (Gainesville: University Press of Florida, 2000); G. L. Simmons, Saudi-Arabia: The Shape of a Client Feudalism (New York: Palgrave, 1999); N. Safran, Saudi Arabia: The Senseless Quest for Security (Cambridge, MA: Belknap Press of Harvard University Press, 1985).

  28 S. Sagan and K. Waltz, The Spread of Nuclear Weapons: A Debate (New York: W. W. Norton, 1995).

  29 For a rich analysis of systems thinking in world politics, see R. Jervis, Systems Effects (Princeton, NJ: Princeton University Press, 1997).

  30 See Fischhoff, “Hindsight,” for the classic experimental demonstration of the effect.

  31 For perhaps the shortest-lived boomster predictions: Joel Kurtzman and Glenn Rifkin, Radical E: from G. E. to Enron, Lessons on How to Rule the Web (New York: John Wiley & Sons, 2001).

  32 For possible futures of the Ukraine, see P. D’Anieri, Politics and Society in Ukraine (Westview Series on the Post-Soviet Republics) (Boulder, CO: West
view Press, 1999); K. Dawisha and B. Parrott, preface to Democratic Changes and Authoritarian Reactionism in Russia, Ukraine, Belarus and Moldova (Cambridge: Cambridge University Press, 1997); T. Kuzio, R. S. Kravchuk, and P. D’Anieri, eds., State and Institution Building in Ukraine (New York: Palgrave, 2000); P. D’Anieri, Economic Interdependence in Ukranian-Russian Relations (New York: State University of New York Press, 1999); I. Prizel, National Identity and Foreign Policy: Nationalism and Leadership in Poland, Russia, and Ukraine (New York: Cambridge University Press, 1998); G. K. Bertsch and W. C. Potter, eds., Dangerous Weapons, Desperate States: Russia, Belarus, Kazakhstan, and Ukraine (London: Routledge, 1999); A. Wilson, The Ukrainians: Unexpected Nation (New Haven, CT: Yale University Press, 2001); R. Szporluk, Russia, Ukraine and the Breakup of the Soviet Union (Stanford: Hoover Institution, 2001).

  33 See R. A. Packenham, The Dependency Movement: Scholarship and Politics in Development Studies (Cambridge: Harvard University Press, 1992); Martin Kramer, Ivory Towers on Sand: The Failure of Middle Eastern Studies in America (Washington, DC: Washington Institute for Near East Studies, 2001).

  34 Many writers on leadership have reached similar conclusions about the importance of personality-context matches in determining whether leadership styles deliver desired results: D. K. Simonton, Genius Creativity and Leadership (Cambridge: Harvard University Press, 1984); D. K. Simonton, Greatness: Who Makes History and Why (New York: Guilford Press, 1994); B. Kellerman, ed., Political Leadership: A Source Book (Pittsburgh: University of Pittsburgh Press, 1986).

  35 The literature on the disintegration of the Soviet Union includes contributions not just by scholars but by many of the original players who have commented on what they think was or was not possible. In addition to the memoirs of Gorbachev and Ligachev, see Anatoly S. Chernyaev, My Six Years with Gorbachev (University Park, PA: Pennsylvania State University Press, 2000). For more detached commentary, see W. Wohlforth, ed., Witnesses to the End of the Cold War (Baltimore: Johns Hopkins University Press, 1996); R. Garthoff, The Great Transition (Washington, DC: Brookings, 1994); D. Oberdorfer, The Turn (New York: Touchstone, 1992). For the argument that Reagan made the cold war last longer, see R. N. Lebow and J. Stein, We All Lost the Cold War (Princeton, NJ: Princeton University Press, 1994). For an analysis of internal forces contributing to collapse, see Vladislav Zubok, “The Collapse of the Soviet Union: Leadership, Elites, and Legitimacy,” in The Fall of the Great Powers, ed. Geir Lundestad (New York: Oxford University Press, 1994). For perspectives that assign credit to Reagan’s policies, see Fareed Zakharia, “The Reagan Strategy of Containment,” Political Science Quarterly (Fall 1990); Peter Schweizer, Victory: The Reagan Administration’s Secret Strategy That Hastened the Collapse of the Soviet Union (City: Atlantic Monthly Press, 1994); P. Rutland, “Sovietology: Notes for a Post-Mortem,” National Interest (1993): 109–22; for the role of personal relations in revising Soviet perceptions of Reagan, see William D. Jackson, “Soviet Reassessment of Ronald Reagan, 1985–1988,” Political Science Quarterly (Winter 1998–99): 617–44. For arguments about the Bush administration’s missed opportunity of 1989, see Robert Legvold, “Lessons from the Soviet Past,” in Reversing Relations with Former Adversaries: U.S. Foreign Policy after the Cold War, ed. C. R. Nelson and K. Weisbrode (Gainesville: University Press of Florida, 1998), 17–43.

  36 For conflicting views of what could have happened in South Africa and of what might yet happen, see Economist, “The End of the Miracle” (December 13, 1997): 117–19; G. Boynton, Last Days in Cloud Cuckooland: Dispatches from White Africa (New York: Random House, 1999); A. Heribert, F. van Zyl Slabbert, and Kogilan Moodley, Comrades in Business: Post-liberation in South Africa (Ultrecht: International Books, 1998); “South Africa’s Uncertain Future,” Economist 345 (1997): 17–21.

  37 M. M. Weinstein, “The Timid Japanese Banking Bailout Just Might Do the Job,” New York Times, October 22, 1998, C2; “Business in Japan: No More Tears,” Economist (November 27, 1999): 4–18; “Three Futures for Japan: Views from 2020,” Economist (March 21, 1998): 25–28; T. L. Friedman, “Japan’s Nutcracker Suite,” New York Times, April 30, 1999, A31; T. J. Pempel, “Japan’s Search for a New Path,” Current History 97(623) (1998): 431–36; “Reality Hits Japan,” Economist 345 (1997): 15–16; “Japan’s Unhappy Introspection,” The World in 1999, Economist (1998): 33.

  38 For a bleak, but not unusually so, account of the Nigerian economy and polity, see K. Maier, This House has Fallen: Midnight in Nigeria (New York: Public Affairs, 2000). On cross-cultural variation in what constitutes an appropriate exchange, see A. Fiske and P. E. Tetlock, “Taboo Trade-offs: Reactions to Transactions That Transgress Spheres of Justice,” Political Psychology 18 (1997): 255–97.

  39 A sign of the hedgehogs’ intensity of conviction here: in the five-year 1997 follow-up, hedgehogs who judged the disintegration of Nigeria to be the most likely class of possible futures in 1992 were in no mood to back off: “OK, so it did not happen in that time frame but be patient and you are going to see an explosion of ghastly proportions: take the Rwandan genocide and multiply by 20 or, if you are the Eurocentric, take Bosnia and multiply by 200”.

  40 On balancing domestic and foreign policy imperatives, see Peter B. Evans, Harold K. Jacobson, and Robert D. Putnam, eds., Double-edged Diplomacy: International Bargaining and Domestic Politics (Berkeley: University of California Press, 1993).

  41 No forecasters scored the forecasting equivalent of a trifecta: assigning their highest likelihood values to war, to Iraqi rout, and to both Iraq and Saddam Hussein surviving such a military debacle. The conceptual ingredients for each correct prediction could be identified in the sample as a whole, but no single individual had all the necessary mental pieces.

  42 P. Krugman, Currency and Crises (Cambridge: MIT Press, 1992); P. Krugman, Pop Internationalism (Cambridge: MIT Press, 1996); Jagdish Bhagwati, The World Trading System at Risk (Princeton, NJ: Princeton University Press, 1991); Jagdish Bhagwati, Protectionism (Cambridge: MIT Press, 1998).

  43 Friedman, “The Lexus and the Olive Tree.”

  44 “China’s Communism, 50 Years On,” The World in 1999, Economist 31 (1998): 56–58; Current History 96(611) (September 1997), special issue on China; D. Burstein, Big Dragon: China’s Future: What It Means for Business, the Economy, and the Global Order (New York: Simon & Schuster, 1998); Xiaobo Lu, Cadres and Corruption: The Organizational Involution of the Chinese Communist Party (Studies of the East Asian Institute) (Stanford, CA: Stanford University Press, 2000); W. W. Lam, China after Deng Xiaoping: The Power Struggle in Beijing Since Tiananmen (Singapore: John Wiley & Sons, 1995); G. Murray, China: The Next Superpower: Dilemmas in Change and Continuity (New York: St. Martin’s Press, 1998); S. S. Kim, ed., China and the World: Chinese Foreign Policy Faces the New Millennium (Boulder, CO: Westview Press, 1998); Lau Chung-Ming and Shen Jianfa, eds., China Review 2000 (Hong Kong: Chinese University Press, 2001); N. D. Kristof and S. Wudunn, China Wakes: The Struggle for the Soul of a Rising Power (New York: Vintage Books, 1995); T. G. Carpenter and J. A. Dorn, eds., China’s Future (Washington, DC: Cato Institute, 2000); B. Gilley, “Jiang Zemin: On the Right Side of History?” Current History 98(629) (1999): 249–53; P. H. B. Godwin, “China’s Nuclear Forces: An Assessment,” Current History 98(629) (1999): 260–65; M. Yahuda, “China’s Search for a Global Role,” Current History 98(629) (1999): 266–70; E. S. Steinfeld, “Beyond the Transition: China’s Economy at Century’s End,” Current History 98(629) (1999): 271–75; J. Fewsmith, “Jiang Zemin Takes Command,” Current History 97(620) (1999): 250–56; M. M. Pearson, “China’s Emerging Business Class: Democracy’s Harbinger?” Current History 97(620) (1999): 268–72; N. R. Lardy, China’s Unfinished Economic Revolution (Washington, DC: Brookings Institution Press, 1998); R. Bernstein and R. H. Munro, The Coming Conflict with China (New York: Alfred A. Knopf, 1997).

  45 Tesser, “Attitude Polarization,” A. H. Eagly and S. Chaiken, The Psychology of Attitudes (Fort Worth, TX: Harcourt Brace Jovano
vich, 1993); A. H. Eagly and S. Chaiken, The Psychology of Attitudes (Fort Worth, TX: Harcourt Brace Jovanovich, 1993).

  46 J. S. Armstrong, Principles of Forecasting: A Handbook for Researchers and Practitioners (Boston: Kluwers, 2001).

  47 This argument foreshadows one of the last-ditch defenses of hedgehogs in chapter 6. As one would expect if foxes were already doing intuitively what averaging does statistically, and what hedgehogs were failing to do (blending perspectives with nonredundant predictive power), hedgehogs benefit more from averaging: the average hedgehog forecast surpasses the average hedgehog forecaster by a far greater margin than the average fox forecast surpasses the average fox forecaster.

  48 G. Gigerenzer and P. M. Todd, Simple Heuristics That Make Us Smart (New York: Oxford University Press, 2000); P. Suedfeld and P. E. Tetlock, “Psychological Advice about Political Decision Making: Heuristics, Biases, and Cognitive Defects,” in Psychology and Social Policy, ed. P. Suedfeld and P. E. Tetlock (Washington, DC: Hemisphere, 1991).

  49 For an expanded discussion of individual differences among executive decision makers on the relative utility of more versus less complex decision procedures, see P. E. Tetlock, “Cognitive Biases and Organizational Correctives: Do Both Disease and Cure Depend on the Politics of the Beholder?” Administrative Science Quarterly 45 (2000): 293–326.

  50 Chapter 3 challenges the influential but often overstated argument that fast-and-frugal heuristics are adaptively superior to more time- and effort-consuming methods of making up our minds. The evidence, reviewed in Suedfeld and Tetlock “Psychology and Social Advocacy,” and Gigerenzer and Todd “Simple Heuristics,” was never sufficient to sustain more than the weak claim that simple decision rules can—under some conditions—produce outcomes as good as, or better than, complex decision rules. Moreover, there are good reasons for supposing that politics poses an especially tough test of fast-and-frugal heuristics. Political observers often latch onto simple heuristics that point in opposite predictive directions. This may be why—as we discover in chapter 6—weighted averages of forecasts (an inherently complex strategy) typically perform better than the majority of individual forecasters (especially the hedgehog extremists among them). Chapter 3 also challenges over-stated claims that overconfidence is an artifact of either regression toward the mean or biased sampling of questions (G. Gigerenzer, “Fast and Frugal Heuristics,” in Blackwell Handbook of Judgment and Decision Making, ed. D. Koehler and N. Harvey [Oxford: Blackwell, 2004]). We tested the former claim and found it wanting. But we could not decisively rule out the latter, more elusive, claim. No one, frankly, knows how to sample questions in an unbiased fashion from the conceptual universes of issues covered in our forecasting exercises. Virtually everyone, however, knows that when the stakes are high enough—the fates of regimes, nations and economies hang in the balance—it is thin consolation to be told that over-confident experts might have done better if we had posed more “representative” questions (see also Chapter 6).