The Theory That Would Not Die

by Sharon Bertsch McGrayne

  Whatever else, the man had panache and fun. Few Harvard professors invited research assistants for Sunday dinners and bottles of 1938 Clos Vougeot, but Schlaifer did. And few took off an entire month or sabbatical year to relax in Greece or France, as he did with his French-born wife, Geneviève, whom he publicly referred to as Snuggle Buggle.

  Schlaifer was not to the manor born. He was born in Vermillion, South Dakota, in 1914 and grew up near Chicago, in a small town where his father was superintendent of schools. At Amherst College he majored in classics and ancient history and took courses in economics and physics. He enrolled in calculus, his only formal math class, solely to capture a large cash prize for the best student. After graduating Phi Beta Kappa at the age of 19, he studied in Athens at the American School of Classical Studies between 1937 and 1939 and earned a Ph.D. in ancient history at Harvard in 1940. Over the next few years he published several articles about religious cults and slavery in ancient Greece. Schlaifer was a quick study, and he filled in for Harvard historians, economists, and physicists as they left for defense work during the Second World War.

  Eventually, Schlaifer was assigned to the university’s Underwater Sound Laboratory, where sonar was developed. He and theoretical physicist Edwin Kemble tried to silence submarine torpedo propellers the better to attack German U-boats. Schlaifer understood the scientific issues well enough to solve equations using Marchant or Frieden electromechanical calculators and to turn technical reports into prose. The war gave him a voracious appetite for practical, real-world problems, and he abandoned ancient history.

  After the war Schlaifer’s physics impressed the Harvard Business School enough to hire him to fulfill a departmental obligation: a study of the aircraft engine industry. He turned the low-status assignment into a triumph, a 600page classic on aviation history, Development of Aircraft Engines, Development of Aircraft Fuel.

  Between his war work and the book, Schlaifer acquired a usefully intimidating campus reputation as a physicist, later cemented by his obituary in the New York Times. He was teaching accounting and production when the business school, despite his spectacularly unsuitable training, assigned him to teach statistical quality control. Knowing nothing about statistics, Schlaifer crammed by reading the dominant theoreticians of the day, Fisher, Neyman, and Egon Pearson. Wartime operations research had mathematized problem solving for two common business problems, inventory control and transportation scheduling. But frequentism offered businesses no help when it came to issues like launching a new product or changing a price.

  Schlaifer’s publications later described modest pleas for help from a newsstand owner unsure of how many copies of the Daily Racing Form to stock, and from a wholesaler worried about allocating his ten delivery trucks between two warehouses. With luck, business owners might make the right decisions. But given the uncertainties involved in even these simple problems, Schlaifer wondered how they could ever hope to systematically make the best possible choices. Even if they could collect additional information by sampling or experimentation, would it be worth the cost?

  According to frequentists, objective statistics were synonymous with long-run relative frequency, and probabilities were invalid unless based on repeatable observations. Frequentists dealt with a rich abundance of directly relevant data and took samples to test hypotheses and draw inferences about unknowns. The method worked for repetitive, standardized phenomena such as successive grain crops, genetics, gambling, insurance, and statistical mechanics.

  But business executives often had to make decisions under conditions of extreme uncertainty, without sample data. As Schlaifer realized, “Under uncertainty, the businessman is forced, in effect, to gamble, . . . hoping that he will win but knowing that he may lose.”4 Executives needed a way to assess probabilities without the repeated trials required by frequentist methods. Schlaifer said teaching frequentism made him feel like a jackass. It simply did not address the main problem in business: decision making under uncertainty.

  Thinking through the problem, Schlaifer wondered how executives could make decisions based on no data. Whatever prior information they had about the demand for their product was obviously better than none. From there, Schlaifer got to the problem of how sample data should be used and how much money should be spent getting it. Updating prior information with sample data got him to Bayes’ rule because it could combine subjectively assessed prior probabilities with objectively attained data. It was a fundamental insight that changed his life.

  Schlaifer did not have much mathematics. Innocent of the raging philosophical divide between objectivists and subjectivists, Schlaifer threw away his books and reinvented Bayesian decision theory from scratch. As a self-made statistician working in a business school, he owed nothing to the statistical establishment. And given his iconoclastic zeal, he brazenly challenged giants in the field. Like Savage, Schlaifer was combining uncertainty with economics in order to make decisions. Savage thought Schlaifer was “hot as a pistol, sharp as a knife, clear as a bell, quick as a whip, and as exhausting as a marathon run.”5

  Schlaifer realized that his mathematical competence was almost nil, “of order of magnitude epsilon.”6 To compensate, he worked 75 to 80 hours a week, puffed through four packs of unfiltered cigarettes a day, and traced his thoughts with different colored chalk across the blackboard in his smokefilled office. Single-mindedly pursuing what he thought was relevant to the real world, he would lurch into one theory, back off, try it another way, and race on to yet another. His voice boomed down the hall almost hourly—“Oh, my God!” or “How could I have been so dumb?”—as he overturned one firmly held opinion with another. Ever curious, he demanded the absolute, best possible analyses. He realized he needed mathematical help.

  Hearing about a young closet Bayesian named Howard Raiffa at Columbia University, Schlaifer scouted him out and persuaded Harvard to hire him. For the next seven years Raiffa and Schlaifer collaborated closely. Raiffa went on to become an international negotiator who wielded enormous influence on education, business, the law, and public policy in the United States and abroad. But he always regarded Schlaifer as “the great man. . . . I revered him; I was in awe of him. . . . He was so positive, so certain, so opinionated, but so smart, so smart. . . . [He was] “a man—a real man—who independently discovered Bayesianism, mocked those who didn’t agree with him, and not only theorized and philosophized but applied the approach to real problems.” Raiffa called him “the—not a but the—most important person in my intellectual development.”7

  Both Schlaifer and Raiffa were elite intellectuals, but Schlaifer’s style was arrogant while Raiffa was, in the words of a collaborator, “a sweetheart, a very warm, open, embracing person.”8 Schlaifer was Phi Beta Kappa in the Ivy League; Raiffa attended City College of New York, “the college of choice for poor and middle-income students in New York—I was on the poor side.”9 During the Second World War, a misplaced template for grading the air force’s arithmetic and elementary algebra test flunked Raiffa and doomed the future advisor to U.S. presidents—not to a prestigious research laboratory like the one where Schlaiffer spent the war but to three rounds of basic training and to grunt assignments in cooks-and-bakers school, meteorology, and a radar blind-landing system.

  Anti-Semitism finally determined Raiffa’s career choice. One day he overheard his army sergeants saying they wanted to line up America’s Jews on a beach and use them for target practice. Later, real estate agents in Fort Lauderdale, Florida, refused to find housing for Raiffa and his wife because they were Jewish. When a friend told him that engineering and science also discriminated against Jews, Raiffa was prepared to believe it. Then he learned that insurance actuaries were graded on objective, competitive examinations. Seeking a field where competence counted more than religion, Raiffa enrolled in the University of Michigan’s actuarial program, where Arthur Bailey had studied.

  To Raiffa’s great surprise, he became a superb and “deliriously happy” student who raced th
rough a bachelor’s degree in mathematics, a master’s degree in statistics, and a doctorate in mathematics in six years between 1946 and 1952. “In the year I studied statistics, I don’t think I heard the word ‘Bayes.’ As a way of inference, it was nonexistent. It was all strictly Neyman-Pearson, classical, objectivistic (frequency-based) statistics.”10

  Although Schlaifer had embraced Bayes in one fell swoop, Raiffa inched grudgingly toward its subjectivity. But reading John von Neumann and Oskar Morgenstern’s book Game Theory (1944), he instinctively assessed how others would play in order to determine how he himself should compete: “In my naiveté, without any theory or anything like that. . . . [I began] assessing judgmental probability distributions. I slipped into being a subjectivist without realizing how radically I was behaving. That was the natural thing to do. No big deal.”11

  When Raiffa gave a series of seminars on Abraham Wald’s new book Statistical Decision Functions, he discovered it was full of Bayesian decisionmaking rules for use in a frequentist framework. Independently of Turing and Barnard, Wald had discovered sequential analysis for testing ammunition while working with the Statistical Research Group in the Fire Control Division of the National Defense Research Committee. Though a confirmed frequentist, he sometimes solved problems in a curiously roundabout manner. After inventing a Bayesian prior, he would solve the Bayesian version of his problem and then analyze its frequentist properties. He also said that every good decision procedure is Bayesian and confided to the statistician Hilda von Mises that he was a Bayesian but did not dare say so publicly. His work would have a large impact on many mathematical statisticians and decision theorists, including Raiffa.

  Until Wald’s book appeared, the word “Bayesian” had referred only to Thomas Bayes’ controversial suggestion about equal priors, not to his theorem for solving inverse probability problems. After Wald died in a plane crash in India in 1950, the Columbia University statistics department hired Raiffa to teach Wald’s course. Raiffa kept a day ahead of his students by reading the textbook every night. He was moving gradually toward a viewpoint opposed by almost every statistics department in the country, including Columbia’s. Giving up “scientific” objectivity and embracing subjectivity would not be easy.

  At first, like Schlaifer, Raiffa taught straight frequentism, using the thencanonical Neyman-Pearson theory, tests of hypotheses, confidence intervals, and unbiased estimation. But by 1955, Raiffa, like Schlaifer, no longer believed these concepts were central. Columbia faculty members were auditing Raiffa’s lectures, and his transformation into a closet Bayesian left him a nervous wreck. He did not “come out” because colleagues whom he admired greatly were vociferously opposed to Bayesianism. “Look, Howard, what are you trying to do?” they asked. “Are you trying to introduce squishy judgmental, psychological stuff into something which we think is science?”12

  He and his Columbia colleagues were working on totally different kinds of problems. Empowered by the Second World War, statisticians like Raiffa and Schlaifer were increasingly interested in using statistics not just to analyze data but to make decisions. In contrast, Neyman and Pearson considered the errors associated with various strategies or hypotheses and then decided whether to accept or reject them; they could not infer what to do based on an observed sample outcome without thinking about all the potential sample outcomes that could have occurred but had not. This was Jeffreys’s objection to using frequentism for scientific inference. Raiffa felt the same way for different reasons; he wanted to make decisions tied to “real economic problems; not phony ones.”13

  Raiffa was interested in concrete, one-of-a-kind decisions requiring quick judgments about how much of a product to stock or how to price it. Like Schlaifer at Harvard, he wanted to help businesses resolve uncertainties and use indirectly relevant information. As Raiffa put it, anti-Bayesians “would never—and I mean never—assign probabilities to some such statement as ‘the probability that p falls in the interval from .20 to .30.’”

  Bayesian subjectivists, on the other hand, actually wanted answers expressed in terms of probabilities. They did not want to merely accept or reject a hypothesis. As Raiffa realized, a business owner wanted to be able to say that “on the basis of my formerly held beliefs . . . and of the specific sample outcomes, I now believe there is a .92 probability that is greater than .25.”14

  This was verboten for frequentists, who recognized only those sample outcomes that were “significant at the .05 level.” Raiffa regarded their focus as “a very, very cursory description of the distribution. I wanted my students to think probabilistically about [the entire distribution of p, about] where the uncertain p could be, and then figure out from the decision point of view where the correct action would be. So the whole question of a test of hypotheses seemed to me to be leading students in a wrong direction.”15

  The chasm between the two schools of statistics crystallized for Raiffa when Columbia professors discussed a sociology student named James Coleman. During his oral examination Coleman seemed “confused and fuzzy . . . clearly not of Ph.D. quality.”16 But his professors were adamant that he was otherwise dazzling. Using his new Bayesian perspective, Raiffa argued that the department’s prior opinion of the candidate’s qualities was so positive that a one-hour exam should not substantially alter their views. Pass him, Raiffa urged. Coleman became such an influential sociologist that he appeared on both the cover of Newsweek and page one of the New York Times.

  So far, Raiffa regarded his transformation from a Neyman-Pearsonite to a Bayesian as an intellectual conversion; his emotional conversion was still to come.

  In a campaign to raise the intellectual standards of business schools, the Ford Foundation donated money to Harvard to hire a mathematical statistician in 1957. The university made Raiffa an attractive offer: joint posts in its new department of statistics and in the business school. When the statistics department learned about Raiffa’s conversion to Bayesian ideas, the chair, Frederick Mosteller, seemed tolerant but lukewarm, and another prominent professor, William Cochran, said, “Well, you’ll grow up.”17 At the business school, though, Schlaifer welcomed Raiffa with open arms.

  Schlaifer was “the most opinionated person I ever met,” Raiffa recalled. At first, he did not realize “how wonderful Schlaifer was. . . . I didn’t realize that Schlaifer was as great a man as he was. He was already focused on the real business decision problems, not on testing hypotheses. He said they were getting it wrong.” Then Raiffa corrected himself: “No, no, he didn’t say that. He said they’re getting it wrong for business decisionmaking under uncertainty.”18

  Each morning Raiffa tutored Schlaifer in calculus and linear algebra, vectors, transformations, and the like. The next morning Schlaifer would conjecture new theorems, and the day after that apply what he had learned to a concrete problem. “His mind was receptive, razor-sharp, tenacious, persistent, creative,” Raiffa discovered.19 Both men were workaholics, but Schlaifer worked longer hours than anyone else. As Raiffa recalled, “He was really a fabulous student. . . . He had raw mathematical abilities provided he could see how it might be put to use.”20 The two never referred to journals or books. Everything they did together was self-generated.

  Schlaifer did not know nearly as much statistics as Raiffa, but he was much better read. Raiffa had not studied the great prewar theorists Jeffreys, Fisher, and Egon Pearson. Later, when he discovered Savage’s work, he was amazed at its clarity. Raiffa took the advice of his colleagues and named his Harvard chair for Frank Ramsey without ever having read the young man’s work.

  Writing articles with Schlaifer, Raiffa always produced the first draft. Then Schlaifer “analyzed everything seven ways to Sunday and changed the text endlessly, putting in commas, and taking them out again,” recalled John Pratt, who wrote several important works with them.21 Schlaifer almost refused to let Harvard Business School Press publish one of his books because its editors put quotation marks outside the punctuation instead of inside. (Thus the b
attle raged over “over.” and “over”.)

  “The unsettling part of him,” Raiffa said, “. . . was that he made so much sense—as long as he stayed away from politics!”22 So Raiffa argued statistics with his collaborator but closed his ears when Schlaifer inveighed against the income tax and advocated solving Haiti’s problems by turning all the Haitians in the United States into soldiers and sending them back home.

  To Schlaifer, Bayes’ rule was not just something to use, it was something to believe in—fervently. A true believer, he refused to accept that there might be several ways to approach a problem. He used “sheer brutal insistence and intellectual discourse by finding holes in every argument that Howard used,” recalled his student Arthur Schleifer Jr. “He would show that these alternative ways led to untenable paradoxes. . . . Robert’s approach was that there was this one way to do it, you have to do it this way, and if you do it any other way, I’ll show you you’re wrong.”23

  Raiffa wanted to expose his students to both frequentist and Bayesian methods so that if some read the establishment’s point of view they would not get confused. But Schlaifer regarded this as teaching falsehood. Anyway, he pronounced loftily, “businessmen don’t read the literature.”24

  By 1958 Raiffa had also converted passionately to subjectivism. It seemed obvious that four businesses serving four different markets could use the same information to produce four different nonstatistical priors and four different conclusions. Although this still bothers some scientists and statisticians, others were content to overwhelm their initial nonstatistical prior opinions with large amounts of new statistical information. Just as many Americans remember where they were when they heard about Kennedy’s assassination or the 9/11 attack, so many Bayesians of Raiffa’s generation remember the exact moment when Bayes’ overarching logic suddenly hit them like an irresistible epiphany. Critics began to call Harvard Business School “a Bayesian hothouse.”