a. A key premise of my original forecast—the assumption that the same underlying forces at work five years ago would continue to be influential—was not satisfied.
b. Another premise of my original forecast—that all other things would remain equal and there would be no major shocks from outside the system—was not satisfied.
c. The more carefully you look at what actually happened, the more you appreciate how close we came to obtaining a very different outcome (but for some minor accidents of history, events could have taken a very different course).
d. The more carefully you look at the current situation, the more you appreciate how easily one of the alternative outcomes might yet still occur (it is still premature to say which predictions will turn out to have been right).
e. Forecasting exercises can yield valuable insights into the validity of competing points of view.
f. Politics is inherently unpredictable.
g. Forecasting exercises are profoundly misleading (they assign too much credit to the lucky winners and too much blame to the unlucky losers).
h. Looking back now, I’d say my assessment of the underlying forces shaping events at the time of the original forecast was sound.
i. Looking back now, I’d say the conceptual or theoretical principles I used to generate the original forecast were sound.
j. Looking back now, I’d say it was a good idea to overestimate some probabilities and to underestimate other probabilities.
Finally, experts were asked “posterior probability questions” prefaced by these instructions: “It is sometimes tempting to argue what happened had to happen and that if only we were wiser, we would have judged what happened as inevitable. And this may indeed be true for some events. But not for all: improbable events do sometimes occur. For example, the number that turns up on any given spin of the roulette wheel was, before the wheel was spun, a highly improbable outcome. We are interested in how, with the benefit of hindsight, you classify the events you once predicted. Do you think the ‘subjective probabilities’ you originally assigned to possible futures (conditional on your understanding of the underlying forces at work back then) were essentially correct? If so, just insert the same estimates you assigned in your prior assessment into ‘Your Current Point of View’ boxes. If not, feel free to change the numbers in any way you now consider appropriate as a result of intervening events (subject only to the constraint that the numbers must still sum up to 1.0).”
Scenario I Scenario I
Scenario II Scenario II
Scenario III Scenario III
Other Scenarios (if applicable) Other Scenarios (if applicable)
Confidence Confidence
Must Sum to _____ Must Sum to _____
1.0 1.0
Your prior assessment Your current point of view
(Numbers on left were filled in, except for hindsight studies.)
Experts were also asked: “Given the political outcomes that did occur, how much confidence do you now have in: (a) the correctness of the understanding (you held at the time of the original forecast) of the underlying forces shaping events; (b) the correctness of the major competing point(s) of view you perceived at the time of the original forecast? Recall that a value of 1.0 indicates absolute confidence in a point of view, 50/50 (0.5) indicates no more confidence than you would have about the outcome of the toss of a fair coin, 0 indicates absolute confidence that a point of view is false.”
Confidence in your prior point
of view or theory
Confidence in your prior point
of view (or theory)
Major Alternative position
Major Alternative position
Other Alternatives
(if applicable)
Other Alternatives
(if applicable)
Confidence
Confidence
Must Sum to 1.0 _____
Must Sum to 1.0 _____
YOUR ASSESSMENT AT THE TIME OF ORIGINAL FORECAST
YOUR CURRENT ASSESSMENT OF POINTS OF VIEW
(Numbers on left were filled in except for hindsight studies.)
III. TURNABOUT TESTS TO PROBE FOR DOUBLE STANDARDS (CHAPTER 5)
The goal here was to assess the magnitude and pervasiveness of “epistemological hypocrisy” in the evaluation of evidence bearing on the plausibility of “lessons of history” that liberals or conservatives deem either politically correct or incorrect.
Participants
These experts were drawn from both the forecasters and belief updaters in sections I and II as well as the experts who made retrospective judgments of close-call counterfactuals in section IV. The turnabout-test exercise reported in chapter 5 focused on the former Soviet Union, with data collection “tacked on” to the questions posed in the forecasting and close-call counterfactual exercise so that initial data collection occurred in 1992 and the number of respondents grew gradually through 2001 (now standing at eighty-nine, of whom approximately half qualify as specialists in either Russian history or the Soviet Union and the other half of whom qualify as specialists in national security policy or international relations with substantial familiarity with the former Soviet Union but not possessing in-depth area knowledge and relevant language skills).
Research Procedures and Materials
Respondents received these instructions: “Since the collapse of the Soviet Union, scholars have gained greater access to once carefully guarded secrets of the Soviet state. We want to explore your reactions to some hypothetical scenarios in which research teams working in the Kremlin archives make discoveries that shed new light on some old controversies.”
At this juncture, participants were randomly assigned to the 2 (evidence-tilt) × 2 (methodological precautions) between-subjects conditions of a factorial experiment conducted in 1992–1993. In the liberal-tilt condition, participants were asked “to imagine that a research team has uncovered evidence in Kremlin archives that indicates history could easily have gone down very different paths at three junctures: specifically, there is evidence that Stalinism could have been averted in the late 1920s, that the U.S. missed major opportunities to end the cold war in the mid-1950s in the aftermath of Stalin’s death, and that Reagan brought the world precariously close to a serious escalation of American-Soviet hostilities in the early 1980s.” In the conservative-tilt condition, participants were asked to imagine the discovery of the same types of evidence, but the evidence now indicates that “history could not have gone down a different path at three much-debated junctures in Soviet history: there is evidence that the Soviet Union would have evolved in a ‘Stalinist’ direction even if Stalin had not been present in the late 1920s, that the U.S. did not miss any opportunities to end the cold war in the mid-1950s, and that there was virtually no risk of a serious escalation of American-Soviet hostilities in the early 1980s.” In each case, “the evidence takes the form of notes, letters, and transcripts of meetings of senior Central Committee or Politburo officials that reveal a strong inclination (disinclination) to pursue a different line of policy: more tolerance for private enterprise and political dissent in the late 1920s, dramatic Soviet concessions on Germany and troop strength in Eastern Europe in the mid-1950s, and confrontational Soviet responses to the American defense buildup in the early 1980s.” In the high-research-quality condition, participants are told that the Kremlinological research team was sensitive to the political implications of their findings and took special precautions to check the authenticity of documents, to consider alternative explanations, and to ensure that all interpretations of text were carefully grounded in historical context. The composition of the team also ensured representation of a broad band of perspectives on the former Soviet Union. In the unspecified-research-quality condition, participants received no such assurances, only a summary description of the team’s goals, work, and findings.
After reading about each discovery, participants agreed or disagreed with the following assertions on nine-point sca
les (with the midpoint of 5 always labeled “unsure”):
1. There are compelling reasons for accepting the conclusions that the investigators want to draw from the evidence.
2. There are strong grounds for suspecting that the motives of the research team as a whole may be political rather than scholarly.
3. There are strong grounds for doubting the authenticity of documents “discovered” on this topic (fears of forgeries and plants).
4. There are strong grounds for suspecting that when the documents and relevant texts are placed in full historical context, they will not make the point that it is claimed.
Participants were also encouraged to offer free-response commentaries on both the counterfactuals and the evidence.
IV. CLOSE-CALL COUNTERFACTUAL EXERCISES (CHAPTER 5)
The goal here was to test two principal hypotheses. Experts’ openness to historical counterfactuals was expected to be a function of (a) the degree to which the counterfactual reinforced or undercut favorite ideological or theoretical generalizations; and (b) the degree to which experts valued explanatory closure (“foxes” versus “hedgehogs”). Studies 1 and 2 focus on the first hypothesis; Studies 3–5 test both hypotheses (as well as the theoretical-beliefs-by-cognitive-style interaction hypothesis).
Study 1: Perceptions of Close Calls in Soviet History
PARTICIPANTS
This study was first conducted in 1992 and asked forty-seven specialists on the former Soviet Union (a sample that consisted of M.A.- and Ph.D.-level professionals working in government and academia) to judge the plausibility of seven counterfactuals that explored contested choice points in Soviet history.
RESEARCH PROCEDURES
We presented respondents with the following instructions: “Making sense of the past often requires making ‘counterfactual’ assumptions about what would have happened if history had taken a different turn. But these counterfactual claims—‘If x had worked out differently, then y would have occurred’—often provoke sharp controversy within the scholarly community as well as in the broader political world. We want to explore your reactions to some controversial counterfactual claims. For example: If World War I had not occurred or taken the form it did, the Bolshevik Revolution never would have succeeded.
“Offer your assessment of how realistic the underlined antecedent condition is. Do we have to rewrite a great deal of history to suppose that the underlined antecedent could have become true, or do we need to alter only one or two minor details or coincidences?
Assuming, for sake of argument, that the antecedent condition were true, how likely do you believe it is that the hypothesized consequence would have occurred?
Assuming, for sake of argument, that the hypothesized consequence occurred, how significant would the long-term impact on history have been? (nine-point scale from 1 [very insignificant] to 5 [moderately significant] to 9 [extremely significant]).”
Respondents in Study 1 then made the same three judgments for each six remaining Soviet counterfactuals:
If Lenin had lived ten years longer, the Bolshevik regime would have evolved in a much less repressive direction than it did.
If Stalin had been deposed as head of the Communist Party in the late 1920s, the Soviet Union would have moved to a kinder, gentler form of Communism fifty years earlier than it did.
If Malenkov had prevailed in the post-Stalin succession struggle, the cold war would have thawed in the 1950s.
If Gorbachev had suddenly died at the same time as Chernenko, the Communist Party would have moved in a much more conservative direction than it did.
If Reagan had not adopted so tough a posture toward the Soviet Union in the early 1980s, the Soviets would not have been as accommodating as they were in the late 1980s.
If Gorbachev had timed and planned his reforms more shrewdly, he could have presided over a reformed and democratic federation of the former Soviet republics.
Study 2: Perceptions of Contingency in South African History
PARTICIPANTS
This study was conducted in 1995 and drew on the expertise of twenty-four specialists who had considerable knowledge of South African politics. As in the Soviet case, these respondents judged the plausibility of the antecedent in each counterfactual, the plausibility of each antecedent-consequent linkage, and the plausibility of supposing the effects will be long lasting.
RESEARCH PROCEDURES AND MATERIALS
The counterfactuals included the following:
1. If it were not for the leadership of F. W. de Klerk, the South African government would have persisted with white-minority rule.
2. If it were not for Nelson Mandela’s leadership, the negotiations between the minority white government and the African National Congress never would have been successful.
3. If the United States had not imposed economic sanctions, the South African government would have persisted with white-minority rule.
4. If there had not been overwhelming pressures to reach accommodation—the rapid growth of the black population and the requirements of labor markets—the South African government would have persisted with white-minority rule.
5. If Soviet-style Communism had not collapsed, the National Party would have been much more reluctant to relinquish white-minority control.
Ideology scale. Nine-point scale, anchored at 1 by strongly agree, at 9 by strongly disagree, and at 5 by unsure:
Unregulated capitalism creates unacceptably large inequalities of income.
There is a viable third path between capitalism and Communism.
A good rule of thumb is: the more government interference in the economy, the poorer the people will be.
The most effective way to preserve peace is to maintain a strong defense posture against would-be aggressors.
I sympathize more strongly with liberal than with conservative causes in contemporary politics.
Studies 3, 4, and 5: Rewriting Twentieth-century History
PARTICIPANTS
Participants were drawn from distinct, but overlapping, populations of scholars who specialized in diplomatic and military history, security studies, and international relations. The eighty-seven participants in Studies 3, 4, and 5—which focused on the origins of World War I, the outcomes of World Wars I and II, and the resolution of various cold war conflicts—were randomly sampled from Divisions 18 (International Conflict) and 19 (International Security and Arms Control) of the American Political Science Association (APSA) and from the Society of Diplomatic and Military Historians. In addition to the measures described next, respondents completed a nine-item version of the need-for-closure scale (items 1–9 in section I).
RESEARCH PROCEDURES AND MATERIALS
Covering-law beliefs. The most relevant statements for Study 3 explored beliefs about causal forces often hypothesized to increase the likelihood of war in general and of World War I in particular:
a. International systems with several great powers are no more likely to erupt into war than are those with only two great powers (reverse scored).
b. It is a myth that multiethnic empires are inherently unstable and a threat to world peace (reverse scored).
c. Changes in the international balance of power—induced by differential growth rates in population and economic power—have historically been the greatest threat to world peace.
d. War is most likely when the state of military technology leads decision makers to believe that the side that strikes first will possess a decisive advantage.
The neorealist-balancing items—especially relevant to the counterfactuals assessed in Study 4—were as follows:
a. For all the talk about a new world order, world politics is still essentially anarchic—the strong do what they will and the weak accept what they must.
b. Whenever one state starts to become too powerful, other states find a way of combining forces and preventing it from dominating them.
c. The security policies of states are often driven by morality, not just by ra
tional calculations of the impact of those policies on the balance of power.
d. It is naïve to suppose that the failure of would-be conquerors such as Philip II, Napoleon, and Hitler to achieve lasting dominance in Europe was predetermined by balance-of-power politics—it might just have been an accident.
The robustness-of-nuclear-deterrence items—especially relevant to Study 5—were as follows:
a. For all the talk about the risk of nuclear accidents, the USA and USSR never really came close to nuclear war.
b. Nuclear weapons played a key role in moderating the behavior of both the American and Soviet governments during the cold war.
c. It is unrealistic to assume that leaders working under great stress will always act with great restraint in crises that raise the risk of the use of nuclear weapons.
Beliefs about close-call counterfactuals. These measures assessed endorsements of the covering laws on nine-point agree-disagree scales. The close-call counterfactuals for Study 3 cast doubt on the inevitability of the First World War:
a. If the carriage driver of Archduke Ferdinand had not taken a fateful wrong turn that gave the Serbian assassins a remarkable second chance to carry out their previously botched assassination plot, war would not have broken out in August 1914.
Expert Political Judgment Page 41