The Inkblots

by Damion Searls

  A more systematic problem with Exner’s system, which others had known about but Wood highlighted, was that the total number of responses a person gave on the test skewed many of the other scores. Giving a lot of answers made a person more likely to be found abnormal in other ways—it changed scores and results that were not supposed to have anything to do with how much the test taker liked to talk. Exner’s system had no way to control for these variations.

  Most dramatically, Wood publicized a problem known about since 2001. The credibility of Exner’s 1989 norms had taken a brutal hit when it was revealed that hundreds of the cases used to calculate them were figments of a clerical error. Someone had apparently pressed the wrong button, and 221 records out of the 700-person sample were counted twice, another 221 records not counted at all. Exner seemed to have known about the mistake for at least two years, but he revealed it only in the middle of a paragraph on this page of the fifth edition of his Rorschach Workbook, offering a new set of norms that he claimed was valid this time. Whether or not the norms were vastly different, Wood called this “an error of enormous magnitude” and was appalled at Exner’s cavalier treatment of a dozen years of potentially invalid diagnoses.

  Wood had general criticisms to make, too. He pointed out that many of Exner’s conclusions rested on hundreds of unpublished studies conducted by his own Rorschach Workshops, whose data had never been made available to outsiders and which had rarely been replicated. He accused the Comprehensive System of having a large component of what we might call science theater, with a blizzard of codes and an avalanche of mutually reinforcing publications wowing a generation of clinical psychologists less trained in statistics and a legal profession unaware of the controversies in clinical psychology.

  These relatively technical attacks on Exner’s system were followed, though, by speculation about why psychologists still “clung to the wreck”—explanations that came across as condescending, even demeaning, to professional readers. For example: It’s hard to change one’s mind. When asked about the book’s split tone, James Wood admitted to “exasperation and open-mouthed disbelief at what’s going on in the Rorschach movement.” The aggressive messaging was justified, he and his coauthors felt, in reaction to sixty years of Rorschach true believers dodging and weaving, dismissing or ignoring inconvenient evidence.

  It probably comes as no surprise that practicing assessment psychologists and Rorschach experts almost unanimously disagreed. Several reviews pointed to Wood’s and his colleagues’ own confirmation bias, selective and slanted presentation of information, reliance on anecdotal evidence (which the book itself criticized), and refusal to distinguish between poor clinical practice and inherent weaknesses in the test. They were not necessarily the impartial scientific arbiters they were claiming to be. One representative review called the book “useful and informative” but cautioned that “each and every study cited by the authors must be carefully scrutinized for selective abstraction and bias to see if it is portrayed accurately.” More than one review pointed out that Rose Martelli’s case, while heartbreaking, had little or no relevance to the value of the Rorschach test when used properly: Rose’s responses had been coded wrong and interpreted badly; her lawyer had apparently requested expert reexamination of the test too late.

  Meanwhile, the critics’ call for a moratorium on the Rorschach in the courts went unheeded. Building on Garb’s 1999 article, Wood’s book had ended with a chapter of advice for lawyers, forensic psychologists, plaintiffs, and defendants, called “Objection, Your Honor! Keeping the Rorschach Out of Court.” But a 2005 statement “intended for psychologists, other mental health professionals, educators, attorneys, judges, and administrators” countered by citing numerous studies to reaffirm Exner’s argument from the nineties. It concluded that “the Rorschach Inkblot Test possesses reliability and validity similar to that of other generally accepted personality assessment instruments and its responsible use in personality assessment is appropriate and justified.” While the article was authored by the not entirely neutral Board of Trustees of the Society for Personality Assessment, the fact remained that the test was still in use. It was cited three times more frequently in appellate cases between 1996 and 2005 than in the previous half century (1945–95), and such testimony was criticized less than a fifth as often, with not one instance of the Rorschach being “ridiculed or disparaged by opposing counsel.”

  Ultimately, the task of grappling with the complex controversy around the Rorschach was left to each individual psychologist or lawyer. While Wood doubted that what he called the “Rorschach cult” would suddenly come around, he hoped that the American public would force them to. “Increased public awareness may be the key to ending psychologists’ long infatuation with the Rorschach,” he wrote, and “word is beginning to leak out.”

  The test had reached an impasse, with two divided camps and onlookers resigned to the fact that different people see different things. When John Exner died in February 2006 at age seventy-seven, he must have thought that this would be his legacy.

  The natural choice to be his successor was Gregory Meyer, a Chicagoan thirty-three years younger than he. Meyer’s 1989 dissertation had raised several of the key failings of Exner’s system that would gain prominence in the late nineties. But he had come to improve the test, not to bury it. He began publishing numerous, densely quantitative papers arguing for updates to the system, and in 1997, when Exner set up a “Rorschach Research Council” to decide what adjustments to his system were necessary, prompted by Wood’s earlier articles, Meyer was on it, able to fight the Rorschach’s scientific battles on the critics’ terms.

  Yet Exner left control of the Comprehensive System—the name, the copyrights—to his family, not to anyone in the scientific community. Exner’s widow, Doris, and their children decided that the system had to stay the way it was: after all the decades of reconciliation and revision on Exner’s part, further updates would no longer be incorporated. The phrase “frozen in amber” comes up often when talking about the decision, and the move seemed so bizarre and counterproductive that conspiracy theories have sprung up to explain it. Whatever the rationale, the Comprehensive System now faced precisely the kind of feud it had been created to overcome.

  Meyer would diplomatically minimize any conflict, saying that negotiations with the Exner heirs were long and the final decision was amicable—that it would be “inaccurate” to call it a “schism” or “warring camps.” But a schism is effectively what it was. He and other leading researchers—four of the six members of Exner’s Rorschach Research Council (Meyer, Donald Viglione, Joni Mihura, and Philip Erdberg), along with a forensic psychologist named Robert Erard—felt they had no choice but to split off and create what is now the latest version of the Rorschach test, first published in 2011: the Rorschach Performance Assessment System, or R-PAS.

  Essentially an update outside the boundaries of Exner’s now-frozen system, R-PAS incorporated the new research and made countless other adjustments, large and small, to bring the real-world Rorschach into the twenty-first century. Ongoing edits to the manual are available online. Abbreviations for codes are simplified to make the system easier to learn. Test results are printed out with information displayed graphically, since printers today are more advanced than typewriters—for example, scores are marked along a line and color-coded either green, yellow, red, or black, depending on how many standard deviations they are away from the norm. The system is a product of consensus, not any one person’s the way the Comprehensive System was Exner’s.

  To solve the problem that more or fewer answers skewed the other results, which Meyer’s dissertation had discussed, he and his colleagues proposed a new approach to administering the test. Test takers would now straightforwardly be told: “We want two, maybe three answers.” If you gave one answer or none, the fact would be noted down but you would be prompted for more: “Remember, we want two, maybe three answers.” If you got carried away, you’d be thanked after your f
ourth answer and asked for the card back.

  This meant giving the test taker a subtly different experience than in years past: the test was more of a concrete task, a little less open-ended and mysterious. It marked another step farther away from Rorschach himself, who privileged the open-ended test-taking experience over standardization. For example, Rorschach had argued in 1921 that measuring reaction times with a stopwatch was “not advisable, because doing so alters the attention of the subject and the harmlessness can be lost….Absolutely no pressure should be exerted.” Now constraints on the test and pressure on the test taker were acceptable prices to pay for better statistical validity.

  In general, examiners administered the test in a more up-front way. They were instructed to avoid saying that there’s no right or wrong answer, for instance, since that’s not entirely true and since thinking in those terms might make people emphasize certain answers. The manual’s suggestions for what to tell a curious test taker were noticeably friendlier in tone than Exner’s scripts (quoted on this page):

  How can you get anything meaningful from inkblots?

  We all see the world a bit differently and this task allows us to understand some of how you see things.

  What does it mean to see a…?

  That’s a good question. If you’d like, we can talk about that when we’re done.

  Why am I doing this?

  It helps us get to know you better so that we can help you more.

  Finally, it was time to get realistic about exposure to the inkblots in the internet era. The Separated Parenting Access and Resource Center, or SPARC, a support group primarily for divorced fathers founded in the late nineties, felt that the Rorschach was inappropriate in custody cases. They seem to have been the first to put the inkblots up online, on one of the site’s first web pages, so that members could refuse to take the Rorschach on the grounds that they had already seen the images. The site even discussed specific responses to each card, while disclaiming that these were “not necessarily ‘good’ responses to the Rorschach….We don’t advise anyone to use the sample responses. What we advise is that you DO NOT take a Rorschach test for any reason.”

  SPARC brushed aside ethical complaints from Rorschach proponents as well as legal complaints from the Swiss publisher, which claimed that the images were copyrighted. In fact they were out of copyright, though the term “Rorschach” has been trademarked since 1991 (it is illegal to call something “a Rorschach test” or “Rorschach cards” and sell it). When the inkblots turned up on Wikipedia in 2009, the Swiss publisher e-mailed, “We are assessing legal steps against Wikimedia,” but there was nothing they could do. The New York Times asked, on the front page: “Has Wikipedia Created a Rorschach Cheat Sheet?”

  The inkblots had long been out in the world, of course. Exner’s books were available in libraries or for purchase; so too was Rorschach’s own. The eye chart for DMV vision tests is online too: people can theoretically memorize the sequence of letters and get a driver’s license despite poor eyesight, but in reality this rarely if ever happens. Yet for decades psychologists had tried to keep the inkblots a secret. That battle was now lost.

  The R-PAS manual took a pragmatic approach: “Because the inkblot images are on Wikipedia and other web sites, and also on clothing and household items like mugs and plates,” examiners should know that “simply having previous exposure to the inkblots does not compromise an assessment.” Studies showed that Rorschach results were “reasonably stable over time.” Rorschach himself had used the same inkblots on the same people more than once. Rather than pretending the blots were still secret, examiners should be taught how to recognize if a test taker had been coached about what to say, and how to deal with intentional “response distortion.”

  In a preliminary 2013 study of what this new world of inkblot accessibility might mean, twenty-five people were shown the Rorschach’s Wikipedia page and asked to “fake good”—to try to make their test results more positive. Compared to a control group, the fakers gave fewer answers overall, more of them standard Popular answers, and so several scores were on average more normal. But this raised red flags in the protocol, and controlling for the inflated number of Popular answers largely eliminated the other effects. The study concluded on a tentative note, calling for much more research.

  Alongside relatively cosmetic changes to the Comprehensive System, Joni Mihura, an R-PAS coauthor and former member of the Research Council (who married Meyer in 2008), was spearheading a herculean project to go through all of Exner’s variables and every existing study on any of them. As Wood and others had pointed out decades before, you can’t, strictly speaking, ask if a test with multiple metrics is valid, only whether each individual metric is valid. Whether or not Movement responses indicate introversion and whether or not the Suicide Index can predict suicide attempts are two very different questions—and neither one is equivalent to deciding if “the Rorschach works.” Since most research considered different scores at the same time, the task of combining all the earlier studies was one of dizzying statistical complexity. It took Mihura and her coauthors seven years.

  They isolated each of the sixty-five core Exner variables and threw out the ones with weak or no empirical evidence for their validity, or that were valid but redundant—a good third of the total. This was a far more rigorous vetting than other tests such as the MMPI, with hundreds of different scores and scales of its own, had ever been subjected to. The variables that passed Mihura’s meta-analysis were the ones accepted into R-PAS; unlike everyone else in the history of Rorschach systems, the creators of R-PAS added no new and untested variables of their own.

  In 2013, Mihura’s findings appeared in Psychological Bulletin, the top review journal in psychology, which hadn’t published on the Rorschach in decades. Her work stood out from the avalanche of other articles and rebuttals, opinions and counterarguments: it put the Rorschach test on a truly scientific footing. And with that, the existential struggle with Wood and the other leading detractors seems to have come to an end. The critics called Mihura’s work “an unbiased and trustworthy summary of the published literature” and officially lifted their call for a moratorium on the Rorschach in clinical and forensic settings “in light of the compelling evidence laid out” by the article, recommending that, yes, the test can be used to measure thought disorders and cognitive processing. The Rorschach had won; many of Wood’s criticisms were addressed, so in a sense the critics won too.

  After building a better Rorschach, the R-PAS creators then had to get people to use it. Mihura’s article gave the baseline: shortly before the introduction of R-PAS, 96 percent of Rorschach clinicians were using Exner’s system. Since then, R-PAS has made headway, but slowly; it will probably prevail, just as Exner’s system eventually did over Klopfer’s and Beck’s, but it hasn’t yet. Most psychologists outside of the theoretical vanguard seem to be sticking with Exner for now; many of them, busy in their practices and not necessarily following the latest research, have never heard of R-PAS. Forensic psychologists have largely stuck with Exner, whether or not they should, since it has years of precedent; three of the R-PAS creators have made the legal case for the new system, but it does not seem to have penetrated far into actual practice yet.

  The conceptual differences between the systems may be relatively minor, but in concrete terms the problems of the era before Exner’s synthesis have returned. Professors have to choose which system to teach, or else teach both and spend less time on either one. As of 2015, more than 80 percent of doctoral programs offering courses on the Rorschach taught Exner, and just over half taught R-PAS. Exner is still what students are most likely to need to know; R-PAS is finding favor in some internship and clinical settings but not all. Research conducted using one system may or may not remain valid when carried over to the other.

  The compromise of R-PAS, like that of Exner’s system before it, was to try to shrink the test down to what could be proven with ironclad validity. This narrowed the te
rms of debate to the point where both sides could agree but may have narrowed the test in other ways as well. Another approach would be to open the test back up, not by making sweeping unscientific claims for its magic X-ray powers but by reconnecting it to a fuller sense of the self, putting it back into the wider world. The test could be revitalized by reimagining altogether what it might be used to do.

  —

  Dr. Stephen Finn, based in Austin, Texas, seems like central casting’s idea of a sensitive psychotherapist: gentle face, white beard, wide open eyes, earnest soft voice. Today, when assessment is most often used to label people for others to then treat, young assessment psychologists admire Finn like no one else in the field—thrilled at the prospect of using their skills in a less secondary role. With his approach, they get to ask not an impartial “What is this person’s diagnosis?” but “What do you want to know about yourself?” Or, even more directly: “How can I help?”

  The set of practices Finn has worked out since the midnineties is known as Collaborative/Therapeutic Assessment, or C/TA. Collaborative assessment means approaching the test session in a spirit of respect, compassion, and curiosity—with a desire to understand the test taker, not primarily to classify or diagnose. Test takers are generally seen as “clients,” not “patients.” Therapeutic assessment means using the process to help clients directly, not just provide information to other decision makers or service providers in the legal or medical system. Both goals—trying to understand clients and trying to change them—go against what Finn has called the “information-gathering” model, which aims to learn facts in order to label people with a diagnosis, an IQ score, or some other preexisting classification.