What Intelligence Tests Miss

by Keith E Stanovich

  In fact, I feel that if we continue this tendency to label every positive cognitive trait with the term intelligence, that will just add to the inappropriate societal deification of MAMBIT that Sternberg, Gardner, and I are united in deploring. Consider a thought experiment. Imagine that someone objected to the emphasis given to horsepower (engine power) when evaluating automobiles. They feel that horsepower looms too large in people’s thinking. In an attempt to deemphasize horsepower, they then begin to term the other features of the car things like “braking horsepower” and “cornering horsepower” and “comfort horsepower.” Would such a strategy serve to make people less likely to look to engine power as an indicator of the “goodness” of a car? I think not. I think it would instead serve to make more salient just the feature that the person wished to deemphasize. Just as calling “all good car things” horsepower would serve to emphasize engine power, I would argue that calling “all good cognitive things” intelligence will contribute to the deification of MAMBIT.6

  Such a strategy will impede educational efforts to foster other cognitive characteristics. For example, critical thinking skills vanish under broad definitions of intelligence. All critical thinking or rationality assessments become part of intelligence if the latter is conceptualized broadly. And again, intelligence-test producers gain from these broad definitions because people will continue to associate the broad concept of intelligence with these tests. How could they not? The tests carry the label intelligence, and the producers of the tests are not eager to discourage the association with broad theories. For example, it took real chutzpah for David Wechsler to define intelligence in his book as “the aggregate or global capacity of the individual to act purposefully, to think rationally and to deal effectively with his environment” (1958, p. 7) despite authoring an IQ test with his name on it that measured no such thing!

  A Different Strategy: Using Dysrationalia to Tame the Intelligence Concept

  My strategy is different from that of the broad theorists. It is to let MAMBIT carve what it can out of nature in scientific terms, label that intelligence, and restrict intelligence to that. We can tame intelligence in folk psychology by pointing out that there are legitimate scientific terms as well as folk terms for the other valued parts of cognitive life and that some of these are measurable. This strategy uses to advantage a fact of life that many IQ-test critics have lamented—that intelligence tests are not going to change any time soon.7 The tests have the label intelligence and thus MAMBIT will always be dominant in the folk psychology of intelligence. I would argue that it is a mistake to ignore this fact.

  Instead, my strategy is to open up some space for rationality in the lexicon of the mental and, in doing so, tame the intelligence concept. My strategy in proposing dysrationalia was to prevent intelligence from absorbing the concept of rationality—something that IQ tests do not measure. I confine the term intelligence to MAMBIT, a practice having the not inconsiderable advantage of getting usage in line with the real world of measurement and testing. We have coherent and well-operationalized concepts of rational action and belief formation. We have a coherent and well-operationalized concept of MAMBIT. No scientific purpose is served by fusing these concepts, because they are very different. To the contrary, scientific progress is made by differentiating concepts. Dysrationalia, and the fact that it is not rare, highlights the fact that “all good things” (rationality in this case) do not always go with MAMBIT.

  Broad views of intelligence that spread the term over a variety of other constructs are in part motivated by a desire to tame the valuation and prestige of IQ tests. The strategy seems to be to downplay the importance of IQ tests by broadening the definition of intelligence to make them (IQ tests) only a small part of this larger concept—a strategy of dilution. But stretching the intelligence concept by dumping into it other positively valued things will not succeed in breaking the link with IQ tests for two reasons. The first reason is that the effects of the one-hundred-year history of associating MAMBIT with the concept intelligence are not going to be easily attenuated. Second, even in the expanded concept of the broad view, MAMBIT remains the easiest component to measure—and the most measurable component will always end up dominating all other components no matter how broad or encompassing the concept.

  If I am right, then the strategy of the broad theorists ends up giving us the worst of all worlds—an intelligence concept more prestigious than ever (because all kinds of other good things have now been associated with it) and the value of MAMBIT further inflated through its association with the new broadened view of intelligence! More important, short shrift is given to the concept of rationality because it is not separately named (but instead conflated with and lost within the intelligence concept). There is no imperative to actually assess rationality, because its semantic space has been gobbled up by the broadened view of intelligence. It will be even harder than it already is to stress that MAMBIT does not measure rational thinking. Although most people recognize that IQ tests do not encompass all of the important mental faculties, we often act (and talk) as if we have forgotten this fact. Where else does our surprise at smart people doing foolish things come from if not from the implicit assumption that rationality and intelligence should go together? The concept of dysrationalia (and the empirical evidence indicating that the condition is not rare) should help to attenuate our surprise at this phenomenon and to create conceptual space in which we can value abilities at least as important as MAMBIT—abilities to form rational beliefs and to take rational action.

  MAMBIT: The Engine of the Brain without a Driver

  Professional psychologists will immediately recognize my proposal to identify intelligence only as MAMBIT as a version of E. G. Boring’s infamous dictum—and this recognition may cause some of them to balk at my proposal. Boring’s dictum was that we should define intelligence as what the intelligence tests measure. However, what made Boring’s suggestion objectionable was that neither he nor anyone else at the time (1923) knew what the tests measured. Because of this, Boring’s definition of intelligence was truly circular. The situation now is totally different. We now know—from the standpoint of information processing and cognitive neuroscience—what the tests measure.

  Unlike some critics of the traditional intelligence concept, I think there has been some justification in the inertia of the psychometric establishment regarding changes in IQ tests and in the (narrow) intelligence concept itself. Traditional intelligence research is a progressive research program in the sense that philosophers of science use that term. There is every indication that work in the traditional paradigm is carving nature at its joints.8 First, the field has a consensus model in the form of the theory of fluid and crystallized intelligence. Much work has gone into uncovering the cognitive subcomponents of fluid intelligence. We now know that there is substantial overlap in the variance in Gf and the variance in measures of working memory capacity.9 Importantly, the computational features of working memory have also been identified during the same period. The most critical insight has been that the central cognitive function tapped by working memory tasks is cognitive decoupling—the ability to manipulate secondary representations that do not track the world in one-to-one fashion as do primary representations.

  Cognitive decoupling appears to be the central cognitive operation accounting for individual differences in Gf and, because of its role in simulation and hypothetical thinking, cognitive decoupling is a crucial mental capacity. Thus, traditional intelligence tests—and MAMBIT—converge on something important in mental life. They represent the fruits of a scientific research program that is progressively carving nature at an appropriate and important joint.10

  I do not wish to minimize the importance of cognitive decoupling—the central individual difference component of MAMBIT. Decoupling operations help us carry out cognitive reform: the evaluation of our own beliefs and the critique of our own desires. Nevertheless, cognitive decoupling as measured on these tests is still a pr
operty of the algorithmic mind that is assessed under maximal rather than typical conditions. Such measures do not assess how typical it is for a person to engage in decoupling operations. They do not assess the propensity of the reflective mind to use such decoupling abilities for cognitive self-critique. They do not assess the tendency to engage in hypothetical thinking to aid problem solving. The ability to sustain cognitive decoupling does not guarantee rationality of behavior or thought. When we measure Gf we measure a critical aspect of the engine of the brain but not the skill of the driver.

  Intelligence Misdefined as Adaptation

  One type of broad definition of intelligence that has strong imperialist tendencies is represented by those definitions that emphasize intelligence as “adaptation to the environment” like that of Wechsler quoted above. Such definitions appropriate large areas of instrumental rationality into the definition of intelligence. To define intelligence as adaptation to the environment when the best known tests of the construct do not assess any such thing creates tremendous potential for confusion.

  Such confusion is apparent in discussions of the so-called Flynn effect in the study of intelligence. Two decades ago, James Flynn systematically documented what some restandardizations of IQ tests had merely suggested—that IQs were rising over time. Overall IQs seem to have risen about 3 points per decade since about 1930. The gains are larger for Gf than for Gc. Cognitive psychologist Ulric Neisser edited a book commissioned by the American Psychological Association concerned with various explanations for the Flynn effect. The explanations considered were nutrition, urbanization, schooling, television, and preschool home environment, among others.11 Interestingly, in his own chapter in the book, Flynn himself favored none of these explanations. Instead, he believed that the intelligence gains are in some sense not “real.” In short, he believed that there have been IQ gains but not intelligence gains, according to his definition. As evidence for his position, he pointed to the lack of cultural flowering that he felt would result from a true intelligence increase. For him, contrary trends were indicated by the fact that “the number of inventions patented in fact showed a sharp decline over the last generation” (1998, p. 35) and that Who’s Who books of eminent scientists were not bursting at the seams.

  But why one would expect such things from an increase in MAMBIT is unclear. The tests do not measure rationality or creativity—things that might really lead to a cultural explosion of the type that Flynn is looking for. In fact, Flynn had tacitly adopted some sort of environmental adaptation definition of intelligence which MAMBIT does not meet. Thus, what some see as a paradox created by the Flynn effect (that IQ gains over the past generation have not been paralleled by concomitant societal achievements) I see as no paradox at all. It is puzzling only because we find it hard to keep in mind that, although our folk concept of intelligence might include adaptation to the environment, the tests on which the gains have been shown do not measure that at all. The tests measure MAMBIT—cognitive decoupling ability that is a critical mental skill, but one that is only one of three components needed for fully rational thought and behavior. The other two are the mindware and thinking dispositions that foster rational thought. That the Flynn effect is thought to present a puzzle shows how difficult it is not to deify intelligence by broadening the definition of it beyond what the tests measure.

  The Dark Side of the Deification of Intelligence

  Such deification of intelligence can have a truly perverse moral consequence that we often fail to recognize—the denigration of those low in MAMBIT. Such denigration goes back to the very beginnings of psychometrics as an enterprise. Sir Francis Galton would hardly concede that those low in IQ could feel pain: “The discriminative facility of idiots is curiously low; they hardly distinguish between heat and cold, and their sense of pain is so obtuse that some of the more idiotic seem hardly to know what it is. In their dull lives, such pain as can be excited in them may literally be accepted with a welcome surprise” (1883, p. 28).

  Milder and subtler versions of this denigration continue down to the modern day. In 2004 author Michael D’Antonio published a book titled The State Boys Rebellion about the ill treatment of boys in the Walter E. Fernald School for the Feebleminded and how a group of boys residing at the school rebelled against this treatment. Disturbingly, however, reviews of the book tended to focus on the stories of those boys who later were found to have normal IQs. The New York Times Book Review (June 27, 2004) titled its review “A Ledger of Broken Arms: Misdiagnosis and Abuse at a School for the ‘Feebleminded’ in the 1950s.” We might ask what in the world does “misdiagnosis” have to do with the issue of highlighting the ill treatment in these institutions? The implication here is that somehow it was less tragic for those “properly diagnosed”—whatever that may mean in this context. Shades of Galton, and of the dark side of the deification of intelligence, are revealed in the reactions to this book.

  The historical tendency that Robert Sternberg has noted—the “tendency to conflate scores on tests of intelligence with some kind of personal value” (2003b, p. 13)—appears in modern life in many guises. As Sternberg suggests, intelligence has come to signify something like one’s personal essence—some indication of personal worth. The deification of IQ and the denigration of low intelligence is now so complete that people would rather have a high IQ than almost any other physical or mental quality. Note, for example, how the diagnosis of intellectual disability has been reduced by almost half in the last 3–4 decades and the explosion of the incidence of disabilities whose definitions—particularly those definitions aimed at parents—stress the presence of normal IQ during the same period (e.g., learning disabilities, ADHD, Asperger’s syndrome).12 This shift is in part a function of social changes, consumerism infecting diagnostic categories, and ascertainment biases introduced by schools, clinicians, and parents. Many parents, for example, are much more accepting of diagnostic categories that do not have “low IQ” attached. Never mind that the life problems associated with some emotional and behavioral disorders are often much greater than those associated with many forms of moderate/mild intellectual disability. As physician G. Robert Delong notes, “in my work I encounter youngsters whose mental retardation is insignificant as compared with their severe disorders of behavior. Finally, it is the behavioral disorder and not intellectual retardation that is decisive and destructive to family life. This suggests a fundamental flaw in the concept of mental retardation: ‘mental’ life constitutes not only intellectual ability (as measured by IQ tests)” (2004, p. 515).

  This comment leads us to an ironic implication of accepting a broad definition of intelligence. If a broad definition is accepted, particularly one that emphasizes the “adaptation to the environment” criterion, then all of the categories of disability that have exploded in recent years will have to be regarded in a new light. Many cases of emotional disorders, behavioral disorders, and ADHD would also now represent cases of low intelligence, because it is almost the defining feature of these disabilities that they represent poor adaptations to the environment. It is fascinating to speculate about whether some of these categories of disability would have become so popular had the broad theorists won the day several decades ago. Imagine that the behavior of an ADHD child was routinely termed “low intelligence” in folk psychology. A response to the thought experiment might be that we would still notice “some difference” between an ADHD child (or even an emotionally disturbed child) and a child with intellectual disability. If we are tempted to give this response, think about what it means. It means that we can notice and label MAMBIT in folk psychology. As indicated earlier in this chapter, scientific evidence does converge on the conclusion that MAMBIT picks out a class of mental operations of considerable importance. The problem is just that folk psychology values those mental operations—and the tests used to measure them—too much. Gf is a mechanism, not a soul.

  Folk Psychology Can Distinguish Intelligence and Rationality

  Finall
y, my argument is, essentially, that we would value MAMBIT less if we would take care to label the things it is not (rationality) and not let the term intelligence incorporate those other things. I think that folk psychology does now differentiate between rationality and intelligence somewhat, but that folk psychology could be reformed to do this even more.

  My feeling that folk psychology could be reformed to further mark the intelligence/rationality distinction is based on a study I conducted with my longtime colleague Richard West some years ago. We had subjects write, in a quite open-ended manner, about what they thought that intelligence was. That is, we asked them to discourse on the following: “What does it mean to say that a person is thinking or behaving intelligently? That is, explain what you mean when you use the term ’intelligence.’ What are the characteristics of intelligent thinking and behavior?” Replicating earlier studies of the folk psychology of intelligence, we found that students had broad theories of intelligence that often incorporated aspects of rationality. However, the theories of intelligence were somewhat less broad when the subjects had previously been asked to give their folk theories of rationality with the following probes: “What does it mean to say that a person is thinking or behaving rationally? That is, explain what you mean when you use the term ‘rationality.’ What are the characteristics of rational thinking and behavior?”

  Even more convincing was a third part of our questionnaire where, after responding with their personal definitions of intelligence and rationality (half the subjects were asked for the former first and half for the latter), the subjects were asked whether or not they differentiated between intelligence and rationality. Specifically, they were presented with the following probe: “Are rationality and intelligence related? Please explain.” Overwhelmingly, subjects did see a difference between the two—often mentioning the “smart but acting dumb” phenomenon (dysrationalia) that I have discussed. Examples of typical responses are the following: