QUALITATIVE ANALYSIS OF ARGUMENTS
Challenge whether the Conditions for Hypothesis Testing were Fulfilled. Each forecast was conditional on the correctness of the expert’s understanding of the underlying forces at work. One does not need to be a logician to appreciate the complexity of this “conditionality.” Experts are free to affix responsibility for errors on the least ego-threatening mistake they made in sizing up situations. Consider three examples:
1. A panel of prominent political scientists announced in August 2000 that their models of presidential elections foretold the outcome of the upcoming Bush-Gore contest. With confidence estimates ranging from 85 percent to 97 percent, they declared Gore would win—and win big with at least 52 percent and perhaps as much as 60 percent of the vote.10 There was no need to pay attention to campaign trivia or polling blips. The incumbent party could not lose: the economy was too strong, the country too secure, and presidential approval too high. Of course, the modelers were wrong. After the election, we found that, the more sympathetic forecasters were to the modelers, the more they argued that the basic premise of the enterprise was still sound: the models had just been fed misleading macroeconomic numbers. The crumbling of the NASDAQ, and a net loss in personal wealth in 2000, had heralded a slowdown in the heady growth rates of the late 1990s. People were already feeling pain.
2. The Persian-Gulf conflict of 1990–1991 also illustrates how experts can rescue core assumptions by tracing their mistakes to “theoretically trivial” misspecifications of antecedent conditions. Observers who expected Saddam to do the “rational thing” and fracture the American alliance by withdrawing from Kuwait frequently insisted they were right: Saddam did do the rational thing. But the rational response changed with changing circumstances. Preemptive withdrawal had ceased to be a face-saving option for Iraq because the United States had blocked it. The geopolitical definition of rationality that inspired the original forecast had been superseded by a domestic political one. Of course, as in the “guess the number” game in chapter 2, it is unclear where to halt this infinite regress of game-theoretic inferences. Why not suppose that Saddam could have blocked the effort to block a face-saving withdrawal? Indeterminacy arises when we do not know where to draw the boundaries on bounded rationality.
3. More generally, it is common in political debates to hear one side complain that it has been stuck with an idiotic prediction. The “wronged” side insists that they were not mistaken about the efficacy of basic strategies of diplomatic influence or basic instruments of macroeconomic policy. They “merely” failed to anticipate how maladroitly the policy would be implemented. Thus, experts insisted at various points in the 1990s that “if NATO had sent the Serbs the right mix of signals, we could have averted this new bloodshed” or “if Yeltsin had practiced real shock therapy, Russia could have avoided this new bout of hyperinflation.” This belief system defense transforms a conditional forecast (if x is satisfied, then y will occur) into a historical counterfactual (if x had been satisfied, then y would have occurred). Counterfactual history becomes a convenient graveyard for burying embarrassing conditional forecasts.
The Exogenous-shock Defense. All hypothesis testing presupposes a ceteris paribus or “all other things equal” clause. In principle, forecasters can always argue that, although the conditions for activating the forecast were satisfied—their understanding of the underlying forces was correct—key background conditions took on unforeseeably bizarre forms that short-circuited the otherwise reliably deterministic connection between cause and effect.
Theorists also find this defense convenient. It gives them license to explain away unexpected events by attributing them to forces outside the logical scope of their theory. One realist, surprised by how far Gorbachev went in making concessions on arms control issues, commented: “I study interstate relations. I am not a Sovietologist. Has a theory of marriage failed if it predicts that a couple will stay unhappily married, the couple toughs it out for decades, and suddenly the husband has a fatal heart attack? Of course not, the failure lies in not checking with the cardiologist. Well, my failure was not recognizing how sick the Soviet state was.”
In the same vein, some modelers of presidential elections attributed Gore’s defeat to an “out of the blue” variable. As a result of Clinton’s effort to cover up his sexual dalliance with Monica Lewinsky, and his subsequent impeachment, a substantial segment of the public had become more attuned to the moral rather than the economic health of the nation. There is no room in parsimonious models for promiscuous presidents and perjury traps.
Exogenous shocks can range from the ridiculous (e.g., a White House intern flashing her underwear at a president with limited capacity to delay need gratification) to the sublime (e.g., the unusual maturity of a political prisoner soon to become president in his country’s first multiracial election) to the tragic (e.g., the assassination of a prime minister of a bitterly divided nation trying to make peace with a longtime foe). And exogenous shocks need not be measured on the micro scale. They can be big: the crumbling of the Soviet economy or financial panics in East Asia, Mexico, and Wall Street, or droughts, earthquakes, and other natural disasters. In effect, anything that falls outside the expert’s framework can qualify. Of course, once “shocked,” experts can choose either to continue excluding the shock from their models (relegating it to irreducible error variance) or to incorporate it (“messing up” their models but also increasing the future absorptive capacity of those models).
The Close-call Counterfactual Defense (“I was almost Right”). This strategy takes the exogenous-shock defense to its natural extreme by explicitly arguing that, although the predicted outcome did not occur, it “almost did” and would have but for trivial and easily-imagined-undone contingencies. Such “close-call counterfactuals” popped up in several forecasting arenas:
1. Observers of the former Soviet Union who, in 1988, thought the Communist Party could not be driven from power by 1993 or 1998 were especially likely to believe that Kremlin hardliners almost overthrew Gorbachev in the 1991 coup attempt, and they would have if the conspirators had been more resolute and less inebriated, or if key military officers had obeyed orders to kill civilians challenging martial law or if Yeltsin had not acted so bravely.
2. Experts who expected the European Monetary Union to collapse argued that the event almost happened during the currency crises of 1992 and would have but for the principled determination (even obstinacy) of politicians committed to the euro and but for the interventions of sympathetic central bankers. Given the abiding conflict of interest between states that have “solid fundamentals” and those that “resort to accounting gimmicks to shrink their budget deficits,” and given “burbling nationalist resentment” of a single European currency, these experts thought it a “minor miracle” that most European leaders in 1997 were still standing by monetary union, albeit on a loophole-riddled schedule.
3. Observers of the U.S. scene who expected Bush to be reelected in 1992 found it easier to imagine a world in which Clinton never became president than did those who foresaw a Clinton victory. All they needed to do was to posit a more compliant Federal Reserve Board (cutting interest rates earlier in 1991 “as it should have done in a recession anyway”) and a deft campaign of negative advertising aimed at Clinton’s character (“strip off the mask and reveal the rogue underneath—the campaign was too gentlemanly”).
4. Experts who expected Quebec to secede from Canada noted how close the second separatist referendum came to passing (“well within sampling error, if a fraction of a percentage point more of the electorate had voted oui, confederation would have unraveled”) and how a more deftly managed campaign could have tipped the outcome (“if a savvier politician, Bouchard rather than Parizeau, had spearheaded the cause, Quebec would be a nation today”).
5. Experts who expected Saddam Hussein to withdraw from Kuwait after the balance of power had tipped against him often claimed that Saddam would have acted as they pr
edicted if he had only understood the situation as clearly as they did. They were also inclined to trace the tragedy to Saddam’s pathological personality, which predisposed him to act far more recklessly than most heads of state. One expert complained: “Who knows what sets him off? Perhaps he thought it better to be a rooster for a day than a chicken for all eternity. But judging from his record, he could have latched on to another proverb and convinced himself that it was better—like Saladin—to retreat in order to fight another day.” In this opinion, we were just a synaptic connection in Saddam’s brain away from averting that war.
6. Observers of South Africa who expected continued white-minority rule from 1989 to 1994 were especially likely to believe that were it not for the coincidental conjunction of two key individuals—de Klerk and Mandela—in leadership roles, South Africa could easily have gone down the path of increasing repression, polarization, and violence.
7. Observers who viewed Kazakhstan as “several Yugoslavias waiting to erupt into interethnic violence” attributed the nonoccurrence to the shrewdness of the Kazakh leadership as well as to the lack of interest among the current crop of Russian leaders in playing the “ethnic card,” something that could easily change as soon as “Yeltsin’s heart finally gives out” and it becomes politically expedient to champion the cause of diaspora Russians.
8. The beleaguered modelers of presidential elections also raised the close-call defense. At first glance, this defense seems a slam dunk. The election will go down as history’s flukiest, hinging on butterfly ballots, hanging chads, and judicial whims. But the macro modelers who projected decisive or even landslide Gore victories should not get off so easily. They need to invoke potent forces to close several-million-vote gaps between their predictions and reality. To this end, they summoned up a variety of close-call scenarios, including “if Nader had not been so appealing or so narcissistic or so stubborn …,” “if Gore had not been such an abysmal debater …,” “if the election had been a few days later, the trend line would have …,” and “if Clinton had not been so incorrigibly self-indulgent….”
The “Just-off-on-Timing” Defense. This strategy moves us from the realm of counterfactual worlds back into the actual world. Experts often insist that, although the predicted outcome has yet to occur, we just need to be patient: it will eventually. This defense is limited, of course, in its applicability to political games in which the predicted outcome has not yet been irreversibly foreclosed. No one expected Al Gore to take George W. Bush’s place in the White House in 2001. Some deals are done deals. But experts did often argue that a trend they deemed likely has merely been delayed and that Canada still will disintegrate (the Parti Québecois will prevail on its third attempt), that Kazakhstan will ultimately fall into a Yugoslav-style conflagration of interethnic warfare (demagogues will seize on the opportunities for ethnic mobilization that Kazakhstan presents), that the European Monetary Union’s misguided effort to create a common currency will someday end in tears and acrimony (the divergent interests of members will trigger crises that even determined leadership cannot resolve), and that nuclear war will eventually be the tragic fate of South Asia or the Korean peninsula. In effect, these experts admitted that they may have been wrong within my arbitrary time frames but they will be vindicated with the passage of time.
The “Politics is Hopelessly Cloudlike” Defense. Experts also have the philosophical option of arguing that, although all preconditions were satisfied and the predicted outcome never came close to occurring and now never will, this failure should not be held against the forecaster. Forecasting exercises are best viewed as lighthearted diversions of no consequence because everyone knows, or else should know, that politics is inherently indeterminate, more cloudlike than clocklike.11 As Henry Kissinger wryly wrote Daniel Moynihan after the fragmentation of the Soviet Union, “Your crystal ball worked better than mine.”12 Here is a concession that concedes nothing.
A variant of this defense warns of the dangers in “stochastic environments” of anointing false prophets (those who got it right were just “lucky”) and of the perils of hastily rejecting sound points of view (those who got it wrong were just unlucky). Some forecasters who thought they could say quite a bit about the future at the onset of our exercise, who certainly thought they had more to offer than a dart-throwing chimpanzee, sounded like radical skeptics by the end of our exercise—a transformation that raises the suspicion that some of the radical skeptics in chapter 2 were bold forecasters who had yet to recover from recent muggings by reality.
The “I made the Right Mistake” Defense. This sixth strategy concedes error but, rather than minimizing the gap between expectation and reality, it depicts the error as the natural by-product of pursuing the right political priorities. As one conservative defiantly declared, “Overestimating the staying power of the Soviet Communist Party was a wise thing to do, certainly wiser than underestimating it.” Some liberals invoked a mirror image of this defense in defense of International Monetary Fund (IMF) loans to Russia in the 1990s. Much of the money may have been misdirected into Swiss bank accounts but, given the risks of allowing a nuclear superpower to implode financially, issuing the loans was, to paraphrase the number 2 decision maker at the IMF, Stanley Fischer, the “prudent thing to do.”13
Experts resorted to the “we made the right mistake” defense in many policy arenas, including nuclear proliferation, ethnic cleansing, and financial contagion. There is, moreover, a common theme running through these examples. In each case, experts possessed value systems that allowed them to portray “false alarms” of change for the worse as less serious than “misses.” As one expert replied a tad testily, “Crying wolf is the price of vigilance.” In the aftermath of the terrorist assaults on New York and Washington, D.C., in September 2001, this expert added: “Americans now understand that far-fetched threats can suddenly materialize and that if you want safety, you better be paranoid.”
The Low-probability Outcome Just Happened to Happen. Our listing of belief system defenses would be incomplete if we did not acknowledge those hardy souls who insisted that their initially low-likelihood estimate of x was not in error and that we need to come to existential terms with living in a low-likelihood world. This defense overlaps a lot with other defenses, such as exogenous-shock and close-call counterfactual arguments, that portray the observed outcome as fluky. But there is a distinction. One could argue that we wound up inhabiting a low-likelihood world without offering any extenuation (such as the higher-likelihood world almost occurred) or excuse (unpredictable forces blew history off course). Decision makers are constantly spinning the roulette wheel of history, so we should not be astonished when the silver ball stops occasionally not in the black or red slots that make up more than 90 percent of the wheel, but in one of the few green slots.
QUANTITATIVE ANALYSIS OF BELIEF SYSTEM DEFENSES
We now shift from listening to arguments to counting them. Our goal is to document how often various groups of forecasters advanced various categories of arguments in various contexts. The closer we look, the stronger the grounds become for characterizing the arguments in aggregate as belief system defenses that experts deployed in largely self-serving ways to justify refusing to change their minds when they lost reputational bets that they themselves once endorsed.
The case for a self-serving bias in argumentation rests on several lines of evidence:
1. The suspicious selectivity with which forecasters advanced arguments that trivialized earlier reputational bets. Experts were far more likely to endorse such arguments when something unexpected occurred. In thirty-nine of sixty comparisons, t-tests revealed that experts who had just lost reputational bets (their most likely future failed to materialize) endorsed arguments from the list of seven more enthusiastically (p < .05) than experts who had just won such bets. By contrast, experts who had won their bets never showed more enthusiasm for “defensive” cognitions than experts who had just lost them.
2. The linkage be
tween the size of mistakes and the activation of defenses. The psychologic is straightforward: the more confident experts were in the original forecast, the more threatening the disconfirmation and the more motivated experts will be to neutralize the troublesome evidence. All else equal, an expert who in 1988 was 90 percent confident that Soviet hardliners would reassert control between 1988 and 1993 should be more unsettled by intervening events than an expert who attached only slightly more than “guessing” confidence to the same forecast. To test this prediction, we created a composite defensiveness index by summing the six measured belief system defenses. The predicted pattern emerged. Among less accurate forecasters, the correlations between ex ante confidence and the composite defensiveness index are always positive, ranging from 0.26 to 0.42 across domains; among more accurate forecasters, the same correlations hover near zero, between – .05 and + 0.08.
3. The linkage between reliance on defenses and retention of confidence in prior opinions. If belief system defenses cushion the blow of unexpected events, then experts whose most likely scenarios do not materialize but who endorse “defensive” cognitions should retain more confidence in their original forecasts after they learn what happened (ex post confidence). But there should be no such correlation among experts whose forecasts were borne out and who should therefore not have experienced any threat to the core tenets of their belief systems. As predicted, among inaccurate forecasters, the defensiveness index is correlated with ex post confidence across domains (correlations ranging from .29 to .59). By contrast, among accurate forecasters, there is almost no relationship between defensiveness and ex post confidence (correlations again hovering near zero, between – .01 and .06).
4. The linkages among reliance on defenses, failures of belief updating, and cognitive style. We find that (a) the more “losers” resisted revising their prior opinions, the more defenses they embraced (r = .49); (b) hedgehog losers (who resisted changing their minds more than fox losers) embraced roughly twice as many defenses as did fox losers; (c) the larger the gap between the amount of belief change prescribed by Bayes’s theorem and the amount of belief change that losing experts conceded, the more defenses experts endorsed (r = .31); (d) when we control for the frequency with which experts endorse belief system defenses, there ceases to be a relationship between being a hedgehog and being a bad loser of reputational bets.14
Expert Political Judgment Page 24