The Theory That Would Not Die

by Sharon Bertsch McGrayne

  The tide turned as growing numbers of workers joined the American Federation of Labor and as local juries started awarding generous settlements to their disabled peers. At that point employers decided it was cheaper to treat occupational health as a predictable business expense than to trust juries and encourage unionization. In an avalanche of no-fault laws passed between 1911 and 1920 all but eight states began requiring employers to insure their workers immediately against occupational injuries and illness. This was the first, and for decades the only, social insurance in the United States.

  The legislation triggered an emergency. Normally, the price of an insurance premium reflects years of accumulated data about such factors as accident rates, medical costs, wages, industrywide trends, and particulars about individual companies. No such data existed in the United States. Not even the most industrialized states had enough occupational health statistics to price policies for all their industries. The industrial powerhouse of New York State had only enough experience to price policies for machine printers and garment workers; South Carolina had only enough for cotton spinners and weavers; and St. Louis and Milwaukee for beer brewers. In 1909 Nebraska had only 25 small manufacturers of any kind. As an insurance expert wondered, “When will Nebraska be able to determine its pure premium on ‘suspenders without buckles,’ or Rhode Island on ‘butchers’ supplies?’ And yet rates must be quoted for either, and moreover they must be adequate and equitable.”12

  Data from other areas were seldom relevant. Germany had collected accident statistics for 30 years, but its industrial conditions were safer, and because its data were collected nationwide, premiums could be based on industrywide information. But in the United States, data were collected by state, and Massachusetts’s statistics about shoe-and bootmakers were irrelevant to Nevada’s metal mines and their high fatality rates because, as one expert reported, “metal mines are so rare in Massachusetts as snakes in Ireland.”13

  Nevertheless, premiums had to be invented—overnight and out of thin air—for almost every sizeable company in the country. It was a nightmare to keep any mathematically trained statistician awake at night—not that the United States had many. Actuaries were often antagonistic to high-level mathematics, and an official complained that accident and fire premiums were typically priced by untrained clerks who used opinion, what they “euphoniously called underwriting judgment,”14 rather like “woman’s intuition . . . (‘I don’t know why I think so, but I am sure I am right’).”15 Compounding the crisis, each state legislature mandated its own unique insurance system.

  Still, premiums had to be priced accurately: high enough to keep the insurance company solvent for the life of its insurees and individualized enough to reward businesses with good safety records. In an extraordinary feat, Isaac M. Rubinow, a physician and statistician for the American Medical Association, organized by hand the analysis, classification, and tabulation of literally millions of insurance claims, primarily from Europe, as a two-or three-year stopgap until each state could accumulate statistics on its occupational casualties. “Every scrap of information,” he said, must be used.16

  Rubinow called together 11 scientifically minded actuaries and formed the Casualty Actuarial Society in 1914. Only seven were college graduates, but their goal was lofty: to set casualty fire and workers’ compensation insurance on a sound mathematical basis. Rubinow became the organization’s first president but left almost immediately when the insurance industry and the American Medical Association opposed extending social insurance to the sick and aged. Rumors swirled that Rubinow, a Jewish immigrant from Russia, had “socialist tendencies.”17

  Albert Wurts Whitney, a specialist in insurance mathematics from Berkeley, replaced Rubinow on a workers’ compensation committee. Whitney was an alumnus of Beloit College and had no graduate degrees but had taught mathematics and physics at the universities of Chicago, Nebraska, and Michigan. At the University of California in Berkeley he had taught probability for future insurance professionals. Though not as steeped in the original mathematical literature as Molina at Bell Labs, Whitney was familiar with the theorems of both Laplace and Bayes and knew he should use one of them. He understood something else too. The equations were too complicated for the fledgling workers’ compensation movement.

  One afternoon in the spring of 1918, during the First World War, Whitney and his committee worked for hours stripping away every possible mathematical complication and substituting dubious simplifications. They agreed to assume that each business in a particular industrial class (for example, all residential roofers) faced equal risks. They would also consider every actuary equally skilled at supplementing injury data with subjective judgments about “nonstatistical” or “exogenous material,” such as a business owner’s drinking habits. This was Bayes’ rule where industrywide experience was used as the basis for the prior and the local business’s history for the new data. Whitney cautioned, “We know that the [subjective] rates for some classifications are more reliable than for others. [But] it is doubtful whether it is expedient in practice to recognize this fact.”18

  By the end of the afternoon the committee had decided to base the price of a customer’s premium almost exclusively on the experience of the client’s broad classification. Thus a machine shop’s premium could be based on data from other, similar businesses or, if it was large enough, its own experience. Combining data from related businesses concentrated the numbers, pulling them closer to the mean and making them more accurate, a subtle “shrinking” effect that Charles Stein would explain in the 1950s. What remained was a stunningly simple formula that a clerk could compute, an underwriter could understand, and a salesman could explain to his customers. The committee proudly named its creation Credibility.

  For the next 30 years the first social insurance system in the United States relied on this simplified Bayesian system. In a classic understatement an actuary admitted, “Of course [Credibility’s] Z = P/(P+K) is not so great a discovery as E = mc2 nor as unalterably true, but it has made life much easier for insurance men for generations.”19 Some 50 years later statisticians and actuaries would be surprised to discover the Bayesian roots of Credibility.

  Next, Whitney worked out methods for weighting each datum as to its subjective believability. Soon actuaries were adventuring “beyond anything that has been proven mathematically. The only demonstration they can make,” an actuary reported later, “is that, in actual practice, it works.”20

  Skeptical state officials and insurance underwriters sometimes wondered where these strange Credibility figures came from. One insurance commissioner demanded, “You have supported everything else in the filing with actual experience, where is the experience supporting your Credibility factor?”21 Actuaries hastily changed the subject. When Whitney was asked where he got the mathematical principles underlying Credibility, he pointed flippantly to a colleague’s home. “In Michelbacher’s dining room,” he said.

  Credibility theory was a practical American response to a uniquely American problem, and it became a cornerstone of casualty and property insurance. As claims accumulated, actuaries could check the accuracy of their premiums by comparing them with actual claims. In 1922 actuaries gained access to an enormous pool of occupational data compiled by the National Council for Compensation Insurance. As the years passed, practicing actuaries had less and less need to understand the relationship between Credibility and Bayes.

  While the United States was using Bayes’ theorem for business decisions and France was adapting it for the military, eugenics was shifting Bayes’ story back to its birthplace in Great Britain. There Karl Pearson and Ronald Fisher were developing statistics—the mathematics of uncertainties—into the first information science. Early in the twentieth century, as they created new ways to study biology and heredity, theoreticians would change their attitudes toward Bayes’ rule from tepid toleration to outright hostility.

  Karl Pearson (I repeat his first name because his son Egon figures in Ba
yes’ story too) was a zealous atheist, socialist, feminist, Darwinist, Germanophile, and eugenicist. To save the British Empire, he believed the government should encourage the upper middle class to procreate and the poor to abstain. Karl Pearson ruled Britain’s 30-odd statistical theorists for years. In the process he introduced two generations of applied mathematicians to the kind of feuding and professional bullying generally seen only on middleschool playgrounds.

  Contentious, unquenchably ambitious, and craggily determined, Karl Pearson was ambivalent about few things, but Bayes’ rule was one of them. Uniform priors and subjectivity made him nervous. With few other tools available to statisticians, however, he concluded sadly that “the practical man will . . . accept the results of inverse probability of the Bayes-Laplace brand till better are forthcoming.”22 As Keynes said in A Treatise on Probability in 1921, “There is still about it for scientists a smack of astrology, of alchemy.” Four years later the American mathematician Julian L. Coolidge agreed: “We use Bayes’ formula with a sigh, as the only thing available under the circumstances.”23

  Another geneticist, Ronald Aylmer Fisher, eventually contested Karl Pearson’s statistical crown and dealt Bayes’ rule a near-lethal blow. If Bayes’ story were a TV melodrama, it would need a clear-cut villain, and Fisher would probably be the audience’s choice by acclamation.

  He didn’t look the part. Even with thick glasses he could barely see three feet and had to be rescued from oncoming buses. His clothes were so rumpled that his family thought he looked like a tramp; he smoked a pipe even while swimming; and if a conversation bored him, he sometimes removed his false teeth and cleaned them in public.

  Fisher interpreted any question as a personal attack, and even he recognized that his fiery temper was the bane of his existence. A colleague, William Kruskal, described Fisher’s life as “a sequence of scientific fights, often several at a time, at scientific meetings and in scientific papers.”24 In a basically sympathetic rendering of Fisher’s career, the Bayesian theorist Jimmie Savage said he “sometimes published insults that only a saint could entirely forgive. . . . Fisher burned even more than the rest of us . . . to be original, right, important, famous, and respected. And in enormous measure, he achieved all of that, though never enough to bring him peace.”25 Part of Fisher’s frustration may have arisen from the fact that, on many statistical matters, he was correct.

  Fisher was 16 when the family business collapsed. At Cambridge on a scholarship, he became the top mathematics student in his class and, in 1911, the founder and chair of the Cambridge University Eugenics Society. A few years later he solved in one page a problem Karl Pearson had struggled over for years; Pearson thought Fisher’s solution was rubbish and refused to publish it in his prestigious journal Biometrika. The two continued to feud as long as they lived. But in straightening out inconsistencies in Karl Pearson’s work, Fisher pioneered the first comprehensive and rigorous theory of statistics and set it on its mathematical and anti-Bayesian course.

  The enmity between these two volatile men is striking because both were fervent eugenicists who believed that the careful breeding of British supermen and superwomen would improve the human population and the British Empire. To help support his wife and eight children on a subsistence farm, Fisher accepted funds from a controversial source, Charles Darwin’s son Leonard, who, as honorary president of the Eugenics Education Society, advocated the detention of “inferior types, . . . the sexes being kept apart” to prevent them from bearing children.26 In return for his financial help, Fisher published more than 200 reviews in Darwin’s magazine between 1914 and 1934.

  In 1919, few jobs were available in statistics or eugenics but Fisher landed a position analyzing fertilizers at Rothamsted Agricultural Experiment Station. Other statistical pioneers worked in breweries, cotton thread and light bulb factories, and the wool industry. Fisher’s job was analyzing volumes of data compiled over decades about horse manure, chemical fertilizers, crop rotation, rainfall, temperature, and yields. “Raking over the muck-heap,” he called it.27 At first, like Karl Pearson, Fisher used Bayes. But during afternoon teas at Rothamsted soil scientists confronted Fisher with new kinds of literally down-to-earth problems. Fascinated, Fisher worked out better ways to design experiments.

  Over the years he pioneered randomization methods, sampling theory, tests of significance, maximum likelihood estimation, analysis of variance, and experimental design methods. Thanks to Fisher, experimental scientists, who had traditionally ignored statistical methods, learned to incorporate them when designing their projects. As the twentieth century’s statistical magistrate, Fisher often ended lengthy discussions with a one-word verdict: “Randomize.” In 1925 he published a revolutionary manual of new techniques, Statistical Methods for Research Workers. A cookbook of ingenious statistical procedures for nonstatisticians, it turned frequency into the de facto statistical method. His first manual sold 20,000 copies, and a second went through seven editions before Fisher’s death in 1962. His analysis of variance, which tells how to separate the effects of various treatments, has become one of the natural sciences’ most important tools. His significance test and its p-values would be used millions of times even as, over the years, it became increasingly controversial. No one today can discuss statistics—what he called “mathematics applied to observational data”—without using some of Fisher’s vocabulary.28 Many of his ideas were solutions to computational problems caused by the limitations of the era’s desk calculators. Soon statistical departments rang with the music of bells activated by mechanical calculating machines at every step of their Fisherian calculations.

  Fisher himself became a superb geneticist who did mathematical statistics on the side. He filled his house with cats, dogs, and thousands of mice for cross-breeding experiments; he could document each animal’s pedigree for generations. Unlike Bayes, Price, and Laplace, he did not need to supplement inadequate or conflicting observations with hunches or subjective judgments. His experiments produced small data sets and subsets tightly focused to answer a single question with rigorous mathematics. He dealt with few uncertainties or gaps in data and could compare, manipulate, or repeat his experiments as needed. Thanks to the problems he analyzed, Fisher redefined most uncertainties not by their relative probabilities but by their relative frequencies. He brought to fruition Laplace’s frequency-based theories, the methods Laplace himself preferred toward the end of his life.

  After 15 years at Rothamsted, Fisher moved, first, to University College London and then to Cambridge as professor of genetics. Today, statisticians regard him as one of the great minds of the twentieth century and a “mythical aura” surrounds both him and Karl Pearson.29 The aura around Fisher is a trifle tarnished, though. He left his family on the farm with a precarious, barebones allowance. As a colleague wrote, “If only . . . if only . . . RAF had been a nicer man, if only he had taken pains to be clearer and less enigmatic, if only he had not been obsessed with ambition and personal bitternesses. If only. But then we might not have had Fisher’s magnificent achievements.”30

  Lashing out at Bayes’ rule, Fisher called it an “impenetrable jungle” and “a mistake, perhaps the only mistake to which the mathematical world has so deeply committed itself.”31 Equal priors constituted “staggering falsity.”32 “My personal conviction,” he declared, is “that the theory of inverse probability is founded upon an error, and must be wholly rejected.”33 A scholarly statistician, Anders Hald, politely lamented “Fisher’s arrogant style of writing.”34 Although Fisher’s work had many Bayesian elements, he battled Bayes for decades and rendered it virtually taboo among respectable statisticians. His constant readiness to quarrel made it hard for opponents to engage him. Bayesians were not alone in concluding that Fisher adopted some of his positions “simply to avoid agreeing with his opponents.”35

  Driven by the need to deal with uncertainties and save time and money, frequency-based sampling theorists enjoyed a golden age during the 1920s and 1930s.
Fisher liberated scientists to summarize and draw conclusions without having to deal with Bayes’ messy prior prejudices and hunches. And thanks to his insistence on mathematical rigor, statistics was becoming, if not quite “real mathematics,” at least a distinct mathematical discipline, mathematics as applied to data.

  The feud between Karl Pearson and Fisher entered its second generation when Karl’s son Egon became another victim of Fisher’s wrath. Unlike his father, Egon Pearson was a modest, even self-effacing gentleman. At first, like his father and Fisher earlier in their careers, Egon Pearson frequently used Bayes’ rule. In 1925 he published the most extensive exploration of Bayesian methods conducted between Laplace in the 1780s and the 1960s. Using priors for a series of seemingly whimsical experiments, he calculated such probabilities as the fraction of London taxicabs with LX license plates; men smoking pipes on Euston Road; horse-drawn vehicles on Gower Street; chestnut colts born to bay mares; and hounds with fawn-spotted coats. His experiments had a serious purpose, though. He was looking at all sorts of binomial problems, “working backwards” to find “nature’s prior,” one that anyone with a binomial problem could use. He concluded that more data had to be collected, but no one took up the challenge. Instead, Egon Pearson busied himself trying to make Fisher’s work more mathematically rigorous, thereby enraging both Fisher and his father.