What Intelligence Tests Miss

by Keith E Stanovich

  12. On the typical versus optimal/maximal distinction, see Ackerman (1994, 1996; Ackerman and Heggestad, 1997; Ackerman and Kanfer, 2004); see also Cronbach (1949); Matthews, Zeidner, and Roberts (2002).

  13. Various authors discuss thinking dispositions (e.g., Ackerman and Heggestad, 1997; Baron, 1985, 2000; Cacioppo et al., 1996; Dole and Sinatra, 1998; Kruglanski and Webster, 1996; Norris and Ennis, 1989; Perkins, 1995; Schommer, 1990; Stanovich, 1999; Sternberg, 1997c, 2003b; Sternberg and Grigorenko, 1997; Strathman et al., 1994).

  14. One reason for endorsing a tripartite structure is that breakdowns in cognitive functioning in the three kinds of minds manifest very differently. For example, disruptions in algorithmic-level functioning are apparent in general impairments in intellectual ability of the type that cause intellectual disability (what used to be called mental retardation). And these disruptions vary quite continuously. Disruptions to the autonomous mind often reflect damage to cognitive modules that result in very discontinuous cognitive dysfunction such as autism or the agnosias and alexias. They often concern so-called subpersonal functions—micro-processing operations rather than the beliefs and goals of the whole person. In contrast, disorders of the reflective mind concern just that—the goals and large-scale actions of the whole person. Difficulties of the reflective mind are present in many psychiatric disorders (particularly those such as delusions) which involve impairments of rationality (see Bermudez, 2001).

  15. These correlations are summarized in a variety of publications (e.g., Ackerman and Heggestad, 1997; Austin and Deary, 2002; Baron, 1982; Bates and Shieles, 2003; Cacioppo et al., 1996; Eysenck, 1994; Goff and Ackerman, 1992; Kanazawa, 2004; Kokis, Macpherson, Toplak, West, and Stanovich, 2002; Noftle and Robins, 2007; Reiss and Reiss, 2004; Zeidner and Matthews, 2000). Furthermore, the correlations that do occur are more often with Gc than with Gf (Ackerman and Heggestad, 1997; Matthews et al., 2002)

  16. On thinking dispositions and the calibration of ambiguous evidence, see Kardash and Scholes (1996) and Schommer (1990). Our argument evaluation task is described in several publications (Stanovich and West, 1997, 1998c; sá, West, and Stanovich, 1999).

  17. One type of problem in this genre of research involves having subjects choose between contradictory car purchase recommendations—one from a large-sample survey of car buyers and the other the heartfelt and emotional testimony of a single friend. For other problems using this type of paradigm, see Fong, Krantz, and Nisbett (1986). For individual differences results using this paradigm, see Kokis et al. (2002) and Stanovich and West (1998c).

  18. See Sá and Stanovich (2001), Stanovich (1999), Stanovich and West (2000), and Toplak and Stanovich (2002). See also research from other laboratories (Bruine de Bruin, Parker, and Fischhoff, 2007; Parker and Fischhoff, 2005).

  19. On the study of self-discipline, see Duckworth and Seligman (2005). There have been many studies linking conscientiousness to important outcome variables (Goff and Ackerman, 1992; Higgins et al., 2007; Ozer and Benet-Martinez, 2006). Tetlock’s work is described in his book Expert Political Judgment (2005). On the study of poor decision-making outcomes, see Bruine de Bruin et al. (2007).

  20. However, working in the other direction (to attenuate the magnitude of the correlations observed in the literature) is the fact that most studies had a restricted range of intelligence in their samples.

  21. See studies by Klaczynski (1997; Klaczynski and Gordon, 1996; Klaczynski, Gordon, and Fauth, 1997; Klaczynski and Lavallee, 2005; Klaczynski and Robinson, 2000). The studies from my lab are reported in several papers (Macpherson and Stanovich, 2007; Sá, Kelley, ho, and Stanovich, 2005; Toplak and Stanovich, 2003). On informal reasoning more generally, see the work of Kuhn (1991, 2005) and Perkins (1985; Perkins et al., 1991).

  22. For a discussion of the term mindware see Perkins (1995).

  23. There are many sources (many written by conservative commentators) that converge in their characterization of Bush’s mental tendencies (Barnes, 2006; Draper, 2007; Frum, 2003; Kessler, 2004; Suskind, 2006; Thomas and Wolffe, 2005; Will, 2005; Woodward, 2006). The consistency in these reports is overwhelming. For example, David Kay, one of the world’s leading experts on weapons inspections, gave a briefing to Bush at the height of the controversy about whether there were weapons of mass destruction in Iraq. Reporter Bob Woodward relates that “Kay left the meeting almost shocked at Bush’s lack of inquisitiveness” (p. 237). Ron Suskind, in his book on America’s security concerns after September 11, reports senior White House staff members worrying “was he reading the materials, was he thinking things through? [italics in original]. . . . left unfettered, and unchallenged, were his instincts, his ‘gut,’ as he often says, and an unwieldy aggressiveness that he’d long been cautioned to contain” (pp. 72–73). This reliance on instincts and gut feelings Over knowledge, information, and thought is a recurring theme among those who know Bush, even among his stoutest defenders. For example, Bob Woodward relates that in an August 20, 2002, interview, Bush himself mentioned instincts as his guides to decision making literally dozens of times. At one point, Bush said to Woodward, “I’m not a textbook player, I’m a gut player” (p. 11).

  Bush is famously incapable of self-criticism. In an April 13, 2004, presidential press conference, a questioner asked him: “In the last campaign, you were asked a question about the biggest mistake you’d made in your life, and you used to like to joke that it was trading Sammy Sosa. You’ve looked back before 9–11 for what mistakes might have been made. After 9–11, what would your biggest mistake be, would you say, and what lessons have [you] learned from it?” Bush’s reply betrayed his problems with counterfactual thinking, his overconfidence, and his reluctance to self-examine. He told the questioner, “I wish you’d have given me this written question ahead of time so I could plan for it. John, I’m sure historians will look back and say, gosh, he could’ve done it better this way or that way. You know, I just—I’m sure something will pop into my head here in the midst of this press conference, with all the pressure of trying to come up with answer, but it hasn’t yet.”

  There is also consensus among commentators that Bush hates doubt and encourages certainty. General Richard B. Myers, chairman of the Joint Chiefs of Staff during the opening of the Iraq war, reported that “when any doubt started to creep into the small, windowless Situation Room, the president almost stomped it out” (Woodward, 2006, p. 371).

  24. The situation regarding crystallized intelligence—Gc—is probably complex in this case. Generally, Gc is related to Gf (see Schweizer and Koch, 2002). However, Gc is also related to the thinking disposition openness to experience (Ackerman and Heggestad, 1997; Bates and Shieles, 2003), in which clearly, given the commentaries I have just reviewed, Bush would be quite low. It is thought that this disposition toward openness leads people to read and to collect the type of information that makes for high Gc. Bush is low on openness and, consistent with what is known about the correlates of that thinking disposition, he does not read nor is he a compulsive information collector. Quite the opposite. All of this would depress his score on a measure of crystallized intelligence below what would be expected from someone with his fluid intelligence, age, social class, and educational peers. Nonetheless, given his university education and social position, compared to a nationally representative sample that forms the norming group for an IQ test, he might well still be average or slightly above on Gc. Most Americans are quite ill informed (Jacoby, 2008; Whittington, 1991).

  25. Interestingly, biographers of Bush’s father, George H. W. Bush, indicate that he by no means shared the extreme cognitive inflexibility of his son. Unlike his son, George H. W. Bush assembled foreign policy advisers of mixed views and listened to opposing ideas before making decisions (Naftali, 2007). Famously, he did not proceed to Baghdad and occupy Iraq after Operation Desert Storm in 1991. He reversed himself and broke a pledge not to raise taxes when economic conditions changed (which perhaps cost him the subsequent election). His nuanced and
restrained behavior during the collapse of the Soviet Union and the eastern bloc is viewed by historians as aiding international relations (Naftali, 2007).

  4 Cutting Intelligence Down to Size

  1. Broad theories include aspects of functioning that are captured by the vernacular term intelligence whether or not these aspects are actually measured by existing tests of intelligence (Ceci, 1996; Gardner, 1983, 1999, 2006a; Perkins, 1995; Sternberg, 1985, 1988, 1997a, 2003b). Narrow theories, in contrast, confine the concept of intelligence to the set of mental abilities actually tested on existing IQ tests. It is important to note that the issue should not be framed dichotomously as broad theories of intelligence versus narrow theories. This is because, significantly, broad theorists do not agree among themselves. For example, Gardner (1999) warns that “Sternberg and I agree more on our criticism of standard intelligence theory than on the direction that new theoretical work should follow” (p. 101).

  2. For example, I am not averse to Sternberg’s (1988, 1997a, 2003b) concern for disrupting the obsessive societal focus on MAMBIT. We do, however, differ on strategies—he being an advocate of stretching the term intelligence to de-emphasize MAMBIT and my preference being to limit the concept intelligence to MAMBIT in order to highlight other terms already in folk psychology that have languished unnecessarily (rationality). This difference in strategy explains why we have disagreed about the appropriate terminology in which to couch our arguments (Stanovich, 1993a, 1993b, 1994a, 1994b; Sternberg, 1993, 1994). Nevertheless, his concepts of practical intelligence, creative intelligence, and wisdom encompass some of the mental properties that I wish to highlight by my emphasis on rationality (see Stanovich, 2001b; Sternberg, 2001, 2003b). Cognitive psychology has been almost exclusively focused on the algorithmic mind and, until quite recently, has given short shrift to the reflective mind. However, Sternberg has been among a handful of investigators (e.g., Ackerman and Heggestad, 1997; Baron, 1982, 1985; Keating, 1990; Moshman, 1994, 2004; Perkins, 1995; Perkins, Jay, and Tishman, 1993; Perkins and Ritchhart, 2004; Stanovich, 1999) who have emphasized concepts such as thinking dispositions (see Sternberg, 1997c; Sternberg and Grigorenko, 1997; Sternberg and Ruzgis, 1994). By and large, psychometric instruments such as IQ tests have tapped cognitive capacities almost exclusively and have ignored cognitive styles, thinking dispositions, and wisdom. Importantly, Baron (1988) argues that, in ignoring dispositions, the IQ concept “has distorted our understanding of thinking. It has encouraged us to believe that the only general determinants of good thinking are capacities, and this attitude has led to the neglect of general dispositions” (p. 122)—a point Sternberg has emphasized in many of his own writings (e.g., Sternberg, 1997c, 2001, 2003b).

  3. On the use of the intelligence term as a motivational tool see Bereiter (2002), Klein (1997), and Willingham (2004). The conceptual coherence of concepts such as social intelligence, emotional intelligence, and practical intelligence continues to be debated in the literature (Brody, 2003, 2004; Cherness, Extein, Goleman, and Weissberg, 2006; Gardner and Moran, 2006; Goleman, 1995, 2006; Keating, 1978; Kihlstrom and Cantor, 2000; Klein, 1997, 2003; Matthews et al., 2002; Sternberg, 2003a, 2006; Visser, Ashton, and Vernon, 2006; Waterhouse, 2006).

  4. of Course, Gardner (1983, 1999) stresses exactly the opposite and emphasizes the independence of his different “intelligences”—that someone high in logical-mathematical intelligence is not necessarily high in musical intelligence. Gardner (1999) also correctly emphasizes the nonfungibility of the intelligences—that one cannot substitute for the other. However, Willingham (2004) has argued that Gardner’s use of the “intelligence” terminology has encouraged just the opposite view among teachers: “It is also understandable that readers believed that some of the intelligences must be at least partially interchangeable. No one would think that the musically talented child would necessarily be good at math. But refer to the child as possessing ‘high musical intelligence,’ and it’s a short step to the upbeat idea that the mathematics deficit can be circumvented by the intelligence in another area—after all, both are intelligences” (Willingham, 2004, p. 24).

  5. I reiterate here the warning that all broad theories of intelligence are not compatible with each other. For example, Gardner (1999) rejects the concepts of creative intelligence, moral intelligence, and emotional intelligence—types of “intelligences” that are quite popular with some other broad theorists. He goes on to warn that “we cannot hijack the word intelligence so that it becomes all things to all people—the psychometric equivalent of the Holy Grail” (p. 210). But in fact, if we concatenate all of the broad theories that have been proposed by various theorists—with all of their different “intelligences”—under the umbrella term intelligence, we will have encompassed virtually all of mental life. Intelligence will be “everything the brain does”—a vacuous concept.

  6. Broad theorists might argue that there are higher correlations among the features of automobiles than there are among the intelligences they propose. I think the data on this conjecture are not in yet (Klein, 1997; Willingham, 2004), and even if a quantitative difference were obtained, I doubt that it would substantially reduce the force of the thought experiment. The point is that when Gardner (1999) states, “I put forth the intelligences as a new definition of human nature, cognitively speaking” (p. 44), he is adding positive valence to the term intelligence and to its closest associates: MAMBIT and the IQ tests themselves.

  7. I cannot resist an “inside baseball” professional remark here. The field of psychology has for decades been plagued by clinical training programs that have had to be dragged kicking and screaming into the scientific world (Dawes, 1994; Lilienfeld, 2007). I would ask advocates of broad definitions of intelligence if they can really imagine, in the near future, thousands of clinical instructors in hundreds of programs vigorously admonishing their students, who are being taught to administer the Wechsler, in the following words: “Now remember, never ever call this assessment intelligence—instead call it analytic capacity or logical-verbal ability, but never just call it intelligence!” Of course, my point is that hell will freeze over before clinical psychology stops calling MAMBIT intelligence, and this must be understood as another huge inertial force in our profession in addition to the psychometric test industry.

  8. To say that “there is every indication that work in the traditional paradigm is carving nature at its joints” is not to deny Gardner’s (1983, 1999) point that there may be additional ways of carving nature that we have been missing; it is only to stress the progress that has been made within the traditional paradigm. Also, because I am focusing here on progress on the psychometric study of intelligence—individual differences—another caveat is in order. Whereas a cognitive scientist might focus on a host of processes when analyzing performance on a particular task, the focus of the psychometrician will be on the (often) much smaller set of information processing operations where large individual differences arise. So when a psychometrician says that process X is the key process in task Z, he or she means that process X is where most of the individual differences arise from, not that process X is all we need to know to understand how task Z is accomplished. Task Z may require many more information processing operations, but these are of less interest to the psychometrician if they are not sources of individual differences.

  9. The overlap is not 100 percent, but regression weights in structural equation models are on the order of .7–.8 (Kane, Hambrick, and Conway, 2005).

  10. This conclusion is often obscured in introductory presentations of intelligence research to beginning students. Introductory psychology textbooks often present to students the broad versus narrow theory of intelligence controversy—usually with a bias toward the former, because it is easier to present nontechnically. Later in the same chapter, the textbook will often make reference to “how difficult it is to measure something as complex as intelligence.” But, of course, there is an inconsistency here. Intelligence is not difficult to measu
re on the narrow view—it is the broad view that causes the measurement problems. We have not only tests, but laboratory measures as well that index MAMBIT pretty precisely in terms of information processing capabilities. It is a point in favor of the narrow concept that we have a reasonably stable construct of it and ways to reliably measure it.

  11. On the Flynn effect, see Flynn (1984, 1987, 2007) and Neisser (1998). My own view of the Flynn effect is that schooling and modernity in general have increased decontextualizing thinking styles and also the use of language as a decoupling tool (Evans and Over, 2004). These mechanisms represent mindware (like rehearsal strategies in short-term memory) that can increase algorithmic-level functioning—particularly the decoupling operation—by making it less capacity demanding and unnatural. Schooler (1998) explores a similar hypothesis in the Neisser volume (see also Greenfield, 1998; Williams, 1998). Interestingly, in a recent book, Flynn (2007) has altered his earlier position and now views the IQ gains as real—the result of the spread of scientific thinking making hypothetical thought more habitual.

  12. Issues surrounding issues of prevalence can be complex (see Barbaresi et al., 2005; Friend, 2005; Parsell, 2004; Gernsbacher, Dawson, and Goldsmith, 2005; Gordon, Lewandowski, and Keiser, 1999; Kelman and Lester, 1997; Lilienfeld and Arkowitz, 2007). I would, however, insert a couple of additional caveats here. First, studies have indicated that ADHD is in fact associated with somewhat lower than normal intelligence (Barkley, 1998), but this empirical finding is not stressed at all on websites and informational packets directed to parents. Second, the tendency for information directed to the public to stress the high intelligence of individuals with learning disabilities is scientifically unjustified, because if learning disabilities were properly diagnosed, they would be just as prevalent in low-IQ as in high-IQ individuals (Stanovich, 2005; Stuebing et al., 2002).