What Intelligence Tests Miss

by Keith E Stanovich

  18. It should be understood that anthropomorphic descriptions of replicator activity are merely a shorthand that is commonly used in biological writings. So for example the statement “replicators developed protective coatings of protein to ward off attacks” could be more awkwardly stated as “replicators that built vehicles with coatings became more frequent in the population.” I will continue the practice here of using the metaphorical language about replicators having “goals” or “interests” in confidence that the reader understands that this is a shorthand only. Thus, I will follow Dawkins (1976/1989) in “allowing ourselves the licence of talking about genes as if they had conscious aims, always reassuring ourselves that we could translate our sloppy language back into respectable terms if we wanted to” (p. 88). The same is true for memes: memes that make more copies of themselves, copy with greater fidelity, or have greater longevity will leave more copies in future generations.

  19. Of course, it is well known that President George W. Bush is strongly reliant on faith-based mindware (e.g., Woodward, 2006).

  20. On belief traps, see Elster (1999) and Mackie (1996).

  21. I have drawn the information on the prevalence of pseudoscientific beliefs in this section from a variety of sources (Druckman, and Swets, 1988; Eisenberg et al., 1993; Farha and Steward, 2006; Frazier, 1989; Gallup and Newport, 1991; Gilovich, 1991, p. 2; Hines, 2003; Musella, 2005; U.S. Congress, 1984). Percentages vary from survey to survey, but they are substantial in all studies. The study of the Mensa members is reported in Chatillon (1989).

  12 How Many Ways Can Thinking Go Wrong?

  1. See Spearman (1904). On Gf/Gc theory see Geary (2005) and Horn and Noll (1997); and on group factors beyond Gf and Gc, see Carroll (1993).

  2. For lists and taxonomies of heuristics and/or rational thinking errors see Arkes (1991), Baron (2000), Harvey (2007), Larrick (2004), McFadden (1999), and Reyna, Lloyd, and Brainerd (2003).

  3. The closely related ideas that the notion of a focal bias conjoins include Evans, Over, and Handley’s (2003) singularity principle, Johnson-Laird’s (1999, 2005) principle of truth, focusing (Legrenzi, Girotto, and Johnson-Laird, 1993), the effect/effort issues discussed by Sperber, Cara, and Girotto (1995), belief acceptance (Gilbert, 1991), and finally, the focalism issues that have been prominent in the literature on affective forecasting (Kahneman et al., 2006; Schkade and Kahneman, 1998; Wilson et al., 2000).

  4. In short, three different types of “start decoupling” calls go out from the reflective mind: decouple the response primed by the autonomous mind so that it can be overridden; copy and decouple a secondary representation in order to carry out simulation; and decouple serial associative cognition in order to start a new serial chain of associations. The three different decoupling operations carried out by the algorithmic mind map suggestively into the components of executive functioning that have been discussed by Miyake et al. (2000): inhibition of prepotent responses, information updating, and set shifting, respectively.

  5. On the “self” as problematic mindware, see Blackmore (1999) and Dennett (1991, 1995).

  6. Among these might be the gambler’s fallacy (Ayton and Fischer, 2004; Burns and Corpus, 2004; Croson and Sundali, 2005; Nickerson, 2004) and many of the other misunderstandings of probability that have been studied in the heuristics and biases literature. Of course, this example highlights the fact that the line between missing mindware and contaminated mindware may get fuzzy in some cases, and the domain of probabilistic thinking is probably one such case.

  7. See Nisbett and Wilson (1977) and the work on the bias blindspot (Ehrlinger, Gilovich, and Ross, 2005; Pronin, 2006).

  8. I have presented a more exhaustive classification of heuristics and biases tasks in other more technical publications (Stanovich, 2008, 2009).

  9. On attribute substitution, see Kahneman and Frederick (2002).

  10. This burgeoning area of research has been the focus of much creative work in the last decade (Ayton, Pott, and Elwakili, 2007; Gilbert, 2006; Gilbert, Pinel, Wilson, Blumberg, and Wheatley, 2002; Hsee and Hastie, 2006; Kahneman, 1999; Kahneman et al., 2006; Kahneman, Diener, and Schwarz, 1999; Schkade and Kahneman, 1998; Wilson and Gilbert, 2005). Hsee and Hastie (2006) describe focalism in hedonic prediction: “predictors pay too much attention to the central event and overlook context events” (p. 31).

  11. There has been much work on the Iowa Gambling Task (Bechara, Damasio, Damasio, and Anderson, 1994; Bechara, Damasio, Tranel, and Damasio, 2005).

  12. There is empirical evidence for rationality failures of the two different types. Dorsolateral prefrontal damage has been associated with executive functioning difficulties (and/or working memory difficulties) that can be interpreted as the failure to override automatized processes (Dempster and Corkill, 1999; Duncan et al., 1996; Harnishfeger and Bjorklund, 1994; Kane and Engle, 2002; Kimberg, D’esposito, and Farah, 1998; Shallice, 1988). In contrast, ventromedial damage to the prefrontal cortex has been associated with problems in behavioral regulation that are accompanied by affective disruption. Difficulties of the former but not the latter kind are associated with lowered intelligence (see Bechara, Damasio, Tranel, and Anderson, 1998; Damasio, 1994; Duncan et al., 1996).

  13. On the heroin addicts, see Petry, Bickel, and Arnett (1998). Our research is reported in Stanovich, Grunewald, and West (2003). There have been several studies of pathological gamblers (Cavedini et al., 2002; Toplak et al., 2007). There is a burgeoning literature on alexithymia and schizophrenia (Bermudez, 2001; Coltheart and Davies, 2000; Mealey, 1995; Murphy and Stich, 2000; Nichols and stich, 2003).

  It is important to emphasize that the Iowa Gambling Task is deliberately designed so that the large rewards in decks A and B will be overwhelmed by penalties (thus resulting in negative expected value). As Loewenstein et al. (2001) point out, it would be easy to design an experiment with the opposite payoff structure—where the risky choices had a higher payoff (Shiv, Loewenstein, Bechara, Damasio, and Damasio, 2005). Indeed, there are real-world examples of just this structure. If one is investing for the long term, stocks—riskier on a short-term basis—tend to outperform bonds. It is an open question which structure (positive expected value being associated with large variance or negative expected value being associated with large variance) is more common in the real world.

  14. See Frederick (2005). Gilhooly and Murphy (2005) have likewise found modest correlations between intelligence and performance on insight problems of this type (see also Toplak and Stanovich, 2002). Of course, the correlations observed in all of these investigations are attenuated somewhat by restriction of range in the university samples.

  15. For individual differences work on framing using both types of designs see Bruine de Bruin et al. (2007) and Stanovich and West (1998b, 1999, 2008b).

  16. There have been many studies of the Mischel paradigm (Ayduk and Mischel, 2002; Funder and Block, 1989; Mischel and Ebbesen, 1970; Mischel, Shoda, and Rodriguez, 1989; Rodriguez, Mischel, and Shoda, 1989). On data from adults, see Kirby, Winston, and Santiesteban (2005). It should be noted that other investigators interpret the failure to delay in the Mischel paradigm not as a failure of the override function but instead as indicating flawed reward and reward discounting mechanisms in the autonomous mind (e.g., Sonuga-Barke, 2002, 2003). If this alternative interpretation is correct, it reclassifies failure in the Mischel paradigm as an instance of the Mr. Spock problem rather than failure of override.

  17. Austin and Deary (2002).

  18. These correlations are derived from a small set of studies (Kokis et al., 2002; Macpherson and Stanovich, 2007; Stanovich and West, 1998c, 2008b).

  19. These mindware gap correlations are derived from a variety of investigations (Bruine de Bruin et al., 2007; Kokis et al., 2002; Parker and Fischhoff, 2005; Sá et al., 1999; Stanovich and West, 1998c, 1998d, 1999, 2000; Toplak and Stanovich, 2002; West and Stanovich, 2003). Some of these data are from studies of children spanning a wide range of ability. The adult samples emplo
y mostly range-restricted university samples.

  20. on research on financial fraud, see consumer Fraud Research Group (2006).

  13 The Social Benefits of Increasing Human

  Rationality—and Meliorating Irrationality

  1. It is true that in the last decade many corporations have tried to broaden their assessment efforts. But they have turned to instruments such as personality tests and so-called honesty tests—most of which are of questionable reliability and validity (Paul, 2005).

  2. Not surprisingly for a psychological attribute that is roughly 50 percent heritable, intelligence is certainly malleable but not to an unlimited extent (Ceci, 1996; Hunt and Carlson, 2007; Neisser, 1998; Neisser et al., 1996; Nickerson, 2004).

  3. These examples are drawn from a variety of sources (Arkes and Ayton, 1999; Baron, 1998, 2000; Bazerman, Baron, and Shonk, 2001; Camerer, 2000; Chapman and Elstein, 2000; Gigerenzer, 2002; Gilovich, 1991; Groopman, 2007; Hastie and Dawes, 2001; Hilton, 2003; Kahneman and Tversky, 2000; Lichtenstein and Slovic, 2006; Margolis, 1996; Myers, 2002; Reyna and Lloyd, 2006; Sunstein, 2002, 2005; Sunstein and Thaler, 2003; Taleb, 2001, 2007). On the study of hatred, see Sternberg (2005).

  4. See the citations here on police psychics (Hines, 2003; Reiser, Ludwig, Saxe, Wagner, 1979), graphology (Ben-Shakhar, Bar-Hillel, Blui, Ben-Abba, and Flug, 1989; Neter and Ben-Shakhar, 1989), and examples of pseudoscientific beliefs on juries and in financial management (Krantz, 2000; Wilkinson, 1998). Many other publications detail further examples (Shermer, 1997; Stanovich, 2004; Sternberg, 2002b).

  5. On the connection between pseudoscientific beliefs and these thinking attributes there is some evidence, but much of it is indirect (Macpherson and Stanovich, 2007; Nickerson, 1998; Shafir, 1994; Stanovich and West, 1997; Toplak et al., 2007; Waganaar, 1988).

  6. In the perennial war in education between teaching declarative knowledge and teaching strategies, the mindware of rationality declares a truce because it comes from both categories. The tendency toward disjunctive thinking is more of a reasoning strategy, whereas many principles of probabilistic reasoning are more akin to declarative knowledge.

  7. There are several sources on the issue of teaching strategies such as disjunctive reasoning as well as more global critical thinking skills (Adams, 1989; Baron and Brown, 1991; Feehrer and Adams, 1986; Kuhn, 2005; Nickerson, 1988, 2004; Reyna and Farley, 2006; Ritchhart and Perkins, 2005; Swartz and Perkins, 1989).

  8. These studies include Arkes et al. (1988); Koehler (1994); Koriat, Lichtenstein, and Fischhoff (1980); Larrick (2004); Mussweiler, Strack, and Pfeiffer (2000); and Tweney et al. (1980). For complications in the implementation of this strategy, see Sanna and Schwarz (2004, 2006).

  9. The work of Nisbett (1993; Fong et al., 1986; Lehman and Nisbett, 1990), Sedlmeier (1999; Sedlmeier and Gigerenzer, 2001), Leshowitz (Leshowitz, DiCerbo, and okun, 2002; Leshowitz, Jenkens, Heaton, and Bough, 1993; see also Larrick, 2004; Zimmerman, 2007), and Kuhn (2005, 2007) is relevant here.

  10. See the work of Gollwitzer (1999; Gollwitzer and Schaal, 1998).

  11. Language input can serve a rapid, so-called context-fixing function (Clark, 1996) in a connectionist network (see Rumelhart, Smolensky, McClelland, and Hinton, 1986). Where it might take an associationist network dozens of trials and a considerable length of time to abstract a prototype, a linguistic exchange can activate a preexisting prototype in a single discrete communication. Clark (1996) calls this the context-fixing function of recurrent linguistic inputs into a connectionist network. Context fixers are “additional inputs that are given alongside the regular input and that may cause an input that (alone) could not activate an existing prototype to in fact do so” (p. 117). Clark (1996) argues that “linguistic exchanges can be seen as a means of providing fast, highly focused, context-fixing information” (p. 117).

  12. There has been a considerable amount of work on forming mental goals (Heath, Larrick, and Wu, 1999; Locke and Latham, 1991) and on affective forecasting (Gilbert, 2006; Kahneman et al., 2006).

  13. On teaching falsifiability at a low level, see Stanovich (2007). The critical thinking skills necessary to avoid pseudoscience have been much discussed (Lilienfeld et al., 2001; Marek et al., 1998).

  14. Organ donation is discussed in Johnson and Goldstein (2006; see also sunstein and Thaler, 2003).

  15. See Sunstein and Thaler (2003).

  16. Much has been written recently on the legislative and corporate impact of these reforms (Benartzi and Thaler, 2001; Camerer et al., 2003; The Economist, 2006; Quinn, 2008: Sunstein and Thaler, 2003; Thaler and Benartzi, 2004; Wang, 2006). On the negative effects of too much choice, see Schwartz (2004).

  17. Gigerenzer’s work is described in numerous sources (Gigerenzer, 2002; Gigerenzer et al., 2005; Todd and Gigerenzer, 2000, 2007). Several studies have demonstrated a variety of ways of presenting probabilistic information so that the relationship between instance and class is clarified in ways that make processing the information easier (Cosmides and Tooby, 1996; Evans et al., 2000; Gigerenzer, 1996, 2002; Girotto and Gonzalez, 2001; Macchi and Mosconi, 1998; Reyna, 2004; Sloman and Over, 2003; Sloman et al., 2003). The physician example is from Friedman (2005).

  18. There has been work on pre-commitment in the domain of saving money (Thaler and Benartzi, 2004) and in other domains (Ariely and Wertenbroch, 2002).

  19. Rozin has done work on the French paradox (Rozin, Kabnick, Pete, Fischler, and Shields, 2003) and the unit bias (Geier, Rozin, and Doros, 2006).

  20. Of course, the argument about whether intelligence is malleable is vexed by the confusion about whether investigators are assuming a broad or narrow definition—whether it is the malleability of MAMBIT or some broader conception that is being debated. Definitional issues plague the entire debate. Nonetheless, I take it that the malleability of MAMBIT is pretty well demonstrated by the existence of the Flynn effect (1984, 1987, 2007; Neisser, 1998). The rise in intelligence over time is greatest for tests like the Raven which are good measures of one of the fundamental cognitive operations underlying MAMBIT—the ability to decouple representations while ongoing mental activity takes place. On teaching rational thought, see Baron (2000) and Nickerson (2004).

  21. Sternberg (2004).

  22. The many universities that are trying to infuse tests of critical thinking into their institutional assessment might be viewed as trying to construct just such an instrument. However, their attempts to measure critical thinking are often theoretically confused by the failure to relate their critical thinking concept to the literature in cognitive science covered in this book—in short, the failure to situate their critical thinking concept in terms of what is known about both intelligence and rational thought.

  BIBLIOGRAPHY

  Abelson, R. P. (1963). Computer simulation of “hot cognition.” In S. Tomkins & S. Messick (Eds.), Computer simulation of personality: Frontier of psychological theory (277–298). New York: John Wiley.

  Abelson, R. P. (1986). Beliefs are like possessions. Journal of the Theory of Social Behaviour, 16, 223–250.

  Ackerman, P. L. (1988). Determinants of individual differences during skill acquisition: Cognitive abilities and information processing. Journal of Experimental Psychology: General, 3, 288–318.

  Ackerman, P. L. (1994). Intelligence, attention, and learning: Maximal and typical performance. In D. K. Detterman (Ed.), Current topics in human intelligence (Vol. 4) (1–27). Norwood, NJ: Ablex.

  Ackerman, P. L. (1996). A theory of adult development: Process, personality, interests, and knowledge. Intelligence, 22, 227–257.

  Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219–245.

  Ackerman, P. L., & Kanfer, R. (2004). Cognitive, affective, and conative aspects of adult intellect within a typical and maximal performance framework. In D. Y. Dai & R. J. Sternberg (Eds.), Motivation, emotion, and cognition: Integrative perspectives on intellectual funct
ioning and development (119–141). Mahwah, NJ: Erlbaum.

  Adams, M. J. (1989). Thinking skills curricula: Their promise and progress. Educational Psychologist, 24, 25–77.

  Ainslie, G. (1992). Picoeconomics. Cambridge: Cambridge University Press.

  Ainslie, G. (2001). Breakdown of will. Cambridge: Cambridge University Press.

  Ainslie, G. (2005). Precis of Breakdown of will. Behavioral and Brain Sciences, 28, 635–673.

  Allingham, M. (2002). Choice theory. New York: Oxford University Press.

  American Psychiatric Association (1994). Diagnostic and statistical manual of mental disorders (IV). Washington, DC: Author.

  Anderson, J. R. (1990). The adaptive character of thought. Hillsdale, NJ: Erlbaum.

  Anderson, J. R. (1991). Is human cognition adaptive? Behavioral and Brain Sciences, 14, 471–517.

  Anderson, M. (2005). Marrying intelligence and cognition: A developmental view. In R. J. Sternberg & J. E. Pretz (Eds.), Cognition and intelligence (268–287). New York: Cambridge University Press.

  Ariely, D., & Wertenbroch, K. (2002). Procrastination, deadlines, and performance: Self-control by precommitment. Psychological Science, 13, 219–224.