by Ian Ayres
Super Crunching in the code and the UI tests: Omnibus Budget Reconciliation Act of 1989, Pub. L. No. 101–239, § 8015, 103 Stat. 2470(1989); Bruce D. Meyer, “Lessons From the U.S. Unemployment Insurance Experiments,” 33 J. Econ. Literature 91 (1995).
Search-assistance regressions: Peter H. Schuck and Richard J. Zeckhauser, Targeting in Social Programs: Avoiding Bad Bets, Removing Bad Apples (2006).
Alternatives to job-search assistance: Instead of job-search assistance, other states tested whether reemployment bonuses could be effective in shortening the period of unemployment. These bonuses were essentially bribes for people to find work faster. A random group of the unemployed would be paid between $500 and $1,500 (between three and six times the weekly UI benefit) if they could find a job fast. The reemployment bonuses, however, were not generally successful in reducing the government’s overall UI expenditures. The amount spent on the bonuses and administering the program often was larger than the amount saved in shorter unemployment spells. Illinois also tested whether it would be more effective to give the bonus to the employer or to the employee. Marcus Stanley et al., Developing Skills: What We Know About the Impacts of American Employment and Training Programs on Employment, Earnings, and Educational Outcomes, October 1998 (working paper).
Good control groups needed: Susan Rose-Ackerman, “Risk Taking and Reelection: Does Federalism Promote Innovation?” 9 J. Legal Stud. 593 (1980).
Other randomized studies that are impacting real-world decisions: The randomized trial has been especially effective at taking on the most intransigent and entrenched problems, like cocaine addiction. A series of randomized trials has shown that paying cocaine addicts to show up for drug treatment increases the chance that they’ll stay clean. Offering the addicts lotteries is an even cheaper way to induce the same result. Rather than paying addicts a fixed amount for staying clean, participants in the lottery studies (who showed up and provided a clean urine sample) earned the chance to draw a slip of paper from a bowl. The paper would tell the addict the size of a prize varying from $1 to $100 (a randomized test about a randomized lottery). See Todd A. Olmstead et al., “Cost-Effectiveness of Prize-Based Incentives for Stimulant Abusers in Outpatient Psychosocial Treatment Programs,” Drug and Alcohol Dependence (2006), http://dx.doi.org/10.1016/j.drugalcdep.2006.08.012.
MTO testing: Jeffrey R. Kling et al., “Experimental Analysis of Neighborhood Effects,” Econometrica 75.1 (2007).
Examples of researchers utilizing such tests to answer these questions: Alan S. Gerber and Donald P. Green, “The Effects of Canvassing, Direct Mail, and Telephone Contact on Voter Turnout: A Field Experiment,” 94 Am. Pol. Sci. Rev. 653(2000); Yan Chen et al., “Online Fund-Raising Mechanisms: A Field Experiment,” 5 Contributions to Econ. Analysis and Pol’y (2006), http://www.bepress.com/bejeap/ contributions/vol5/iss2/art4; Stephen Ansolabehere and Shanto Iyengar, Going Negative: How Political Advertisements Shrink and Polarize the Electorate (1995).
Testing college roommates: Michael Kremer and Dan M. Levy, “Peer Effects and Alcohol Use Among College Students,” Nat’l Bureau of Econ. Research Working Paper No. 9876 (2003) (men who drank prior to college earned much lower grades if they were placed with a heavy drinker instead of a teetotaler).
Testing ballot order: Daniel E. Ho and Kosuke Imai, “The Impact of Partisan Electoral Regulation: Ballot Effects from the California Alphabet Lottery, 1978–2002,” Princeton L. and Pub. Affairs Paper No. 04–001, Harvard Pub. L. Working Paper No. 89 (2004).
For more information on Joel’s work, see the following smorgasbord of Waldfogel empiricism: Joseph Tracey and Joel Waldfogel, “The Best Business Schools: A Market-Based Approach,” 70 J. Bus. 1 (1997); Joel Waldfogel, “The Deadweight Loss of Christmas: Reply,” 88 Am. Econ. Rev. 1358 (1998); Felix Oberholzer-Gee et al., “Social Learning and Coordination in High-Stakes Games: Evidence from Friend or Foe,” Nat’l Bureau of Econ. Research, Working Paper No. 9805 (2003), http://www.nber.org/papers/W9805; Joel Waldfogel, “Aggregate Inter-Judge Disparity in Federal Sentencing: Evidence from Three Districts,” Fed. Sent’g Rep., Nov.–Dec. 1991.
Other market reactions: Joel and I have also ranked judges’ bail decisions based on how the bail bonds market reacted to the decisions. Ian Ayres and Joel Waldfogel, “A Market Test for Race Discrimination in Bail Setting,” 46 Stan. L. Rev. 987 (1994).
Post-conviction earnings: Jeffrey R. Kling, “Incarceration Length, Employment, and Earnings,” Am. Econ. Rev. (2006).
Recidivism rates: Danton Berube and Donald P. Green, “Do Harsher Sentences Reduce Recidivism? Evidence from a Natural Experiment” (2007), working paper.
The Poverty Action Lab: J. R. Minkel, “Trials for the Poor: Rise of Randomized Trials to Study Antipoverty Programs,” Sci. Am. (Nov. 21, 2005), http://www. sciam.com/article.cfm?articleID=000ECBA5-A101-137B-A10183414B7F0000.
Female chiefs: Raghabendra Chattopadhyay and Esther Duflo, “Women’s Leadership and Policy Decisions: Evidence from a Nationwide Randomized Experiment in India,” Inst. Econ. Dev. Paper No. dp-114 (2001), http://www. bu.edu/econ/ied/dp/papers/chick3.pdf.
Testing teacher absenteeism: Esther Duflo and Rema Hanna, “Monitoring Works: Getting Teachers to Come to School,” Nat’l Bureau of Econ. Research Working Paper No. 11880 (2005); Swaminathan S. A. Aiyar, “Camera Schools: The Way to Go,” Times of India, Mar. 11, 2006, http://timesofindia.india times.com/articleshow/1446353.cms.
Kenyan de-worming: Michael Kremer and Edward Miguel, “Worms: Education and Health Externalities in Kenya,” Poverty Action Lab Paper No. 6(2001), http://www.povertyactionlab.org/papers/kremer_miguel.pdf.
Auditing in Indonesia: Benjamin A. Olken, “Monitoring Corruption: Evidence from a Field Experiment in Indonesia,” Nat’l Bureau of Econ. Research Working Paper No. 11753 (2005); “Digging for Dirt,” Economist, Mar. 16, 2006.
Progresa’s poverty program: Paul Gertler, “Do Conditional Cash Transfers Improve Child Health?: Evidence from PROGRESA’s Control Randomized Experiment,” 94 Am. Econ. Rev. 336 (2004).
Paying mothers vs. fathers: Gertler stressed that targeting mothers may not be appropriate in other cultures. “In fact, in Yemen right now we are doing a conditional cash transfer where we are randomizing giving the money to the mother vs. giving the money to the father to see whether it makes a difference.” See Paul Gertler, “Do Conditional Cash Transfers Improve Child Health?: Evidence from PROGRESA’s Control Randomized Experiment,” 94 Am. Econ. Rev. 336 (2004).
CHAPTER 4
The beginning of EBM: Gordon Guyatt et al., “Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine,” 268 JAMA 2420 (1992); “Glossary of Terms in Evidence-Based Medicine,” Oxford Center for Evidence-Based Medicine, http://www.cebm.net/glossary.asp; Gordon Guyatt et al., “Users’ Guide to the Medical Literature: Evidence-Based Medicine: Principles for Applying the Users’ Guides to Patient Care,” 284 JAMA 1290 (2000).
Ignaz Semmelweis takes the next step in medical Super Crunching: “The Cover: Ignaz Philipp Semmelweis (1818–65),” 7 Emerging Infectious Diseases, cover page (Mar.–Apr. 2001), http://www.cdc.gov/ncidod/eid/vol7no2/cover.htm.
Pasteur also early supporter of hand-washing: Louis Pasteur was, however, convinced by Semmelweis’s results. In a speech before the Academy of Medicine, Pasteur opined: “If I had the honor of being a surgeon…not only would I use none but perfectly clean instruments, but I would clean my hands with the greatest care….” Theodore L. Brown, Science and Authority (book manuscript, 2006).
Don Berwick, the modern-day Semmelweis: Tom Peters, “Wish I Hadn’t Read This Today,” Dispatches from the World of Work, Dec. 7, 2005, http:// tompeters.com/entries.php?note=008407.php; Inst. of Med., To Err Is Human: Building a Safer Health System (1999); Donald Goldmann, “System Failure Versus Personal Accountability—The Case for Clean Hands,” 355 N. Engl. J. Med. 121 (Jul. 13, 2006); Neil Swidey, “The Revolutionary,” Boston Globe, Jan. 4, 2004; Donald M. Berwick et al., “The 100,000 Lives Campaign:
Setting a Goal and a Deadline for Improving Health Care Quality,” 295 JAMA 324 (2006); “To the Editor,” N. Engl. J. Med. (Nov. 10, 2005).
Reducing risk of central-line catheter infections: S. M. Berenholtz, P. J. Pronovost, P. A. Lipsett, et al. “Eliminating Catheter-Related Bloodstream Infections in the Intensive Care Unit,” Crit. Care Med. 32 (2004), pp. 2014–2020.
Medical myths and practices that won’t die: Gina Kolata, “Annual Physical Checkup May Be an Empty Ritual,” N.Y. Times, Aug. 12, 2003, p. F1; Douglas S. Paauw, “Did We Learn Evidence-Based Medicine in Medical School? Some Common Medical Mythology,” 12 J. Am. Bd. Family Practice 143 (1999); Robert J. Flaherty, “Medical Mythology,” http://www.montana.edu/wwwebm/ myths/home.htm.
Can alternative medicine be evidence-based?: The resistance to statistical evidence is even more pronounced with regard to “alternative medicine,” a vast field that encompasses everything from herbal supplements to energy therapy to yoga. More than a third of American adults use some form of alternative medicine annually; they spend more than $40 billion a year on alternative treatments. Patricia M. Barnes, et al., “Complementary and Alternative Medicine Use Among Adults,” CDC Advance Data Report No. 343 (May 27, 2004), available at http://www.cdc.gov/nchs/data/ad/ad343.pdf.
Many adherents and advocates of alternative medicine reject not only Western treatments but the Westernized notion of statistical testing. They sometimes claim that their practices are too “individual” or “holistic” to study scientifically and instead rely on anecdotes and case studies without adequate controls or control groups for comparison. I’m agnostic about whether alternative medicine is effective. But it verges on idiocy to claim that the effectiveness cannot be tested. If it really is important, as alternative medicine advocates claim, to take into account a larger set of information about the patient (“what kind of person has the disease”), then providers who do so should produce better results. There’s no reason why Super Crunching can’t be used to test whether alternative medicine works. To support EBM doesn’t mean that you’re hostile to alternative or innovative—even seemingly wacky—therapies. When it comes to the back-end inquiry of finding out which treatments are effective, there is no East and West. I throw my lot with two past editors-in-chief of the New England Journal of Medicine, Marcia Angell and Jerome Kassirer:
It is time for the scientific community to stop giving alternative medicine a free ride. There cannot be two kinds of medicine—conventional and alternative. There is only medicine that has been adequately tested and medicine that has not, medicine that works and medicine that may or may not work.
M. Angell and J. P. Kassirer, “Alternative Medicine: The Risks of Untested and Unregulated Remedies,” 339 N. Engl. J. Med. 839 (1998). See also Kirstin Borgerson, “Evidence-Based Alternative Medicine?” 48 Perspectives in Bio. and Med. 502 (2005); W. B. Jonas, “Alternative Medicine: Learning from the Past, Examining the Present, Advancing to the Future,” 280 JAMA 1616 (1998); P. B. Fontanarosa and G. D. Lundberg, “Alternative Medicine Meets Science,” 280 JAMA 1618 (1998).
The Aristotelian approach: Kevin Patterson, “What Doctors Don’t Know (Almost Everything),” N.Y. Times Magazine, May 5, 2002; David Leonhardt, “Economix: What Money Doesn’t Buy in Health Care,” N.Y. Times, Dec. 13, 2006.
Contemporary uses (or lack thereof) of EBM: Brandi White, “Making Evidence-Based Medicine Doable in Everyday Practice,” Family Practice Mgt. (Feb.2004), http://www.aafp.org/fpm/20040200/51maki.htm; Jacqueline B. Persons and Aaron T. Beck, “Should Clinicians Rely on Expert Opinion or Empirical Findings?” 4 Am. J. Mgd Care 1051 (1998), http://www.ajmc.com/files/article files/AJMC1998JulPersons1051_1054.pdf; D. G. Covell et al., “Information Needs in Office Practice: Are They Being Met?” 103 Ann. Internal Med. 596 (1985); John W. Ely et al., “Analysis of Questions Asked by Family Doctors Regarding Patient Care,” 319 BMJ 358 (1999); “The Computer Will See You Now,” Economist, Dec. 8, 2005, http://www.microsoft.com/business/peopleready/news/economist/ computer.mspx.
Coffee and Heart Disease: A. Z. LaCroix et al., “Coffee Consumption and the Incidence of Coronary Heart Disease,” 315 N. Engl. J. Med. 977 (1986); Int’l Food Info. Council Foundation, “Caffeine and Health: Clarifying the Controversies”(1993), www.familyhaven.com/health/ir-caffh.htm.
91: 17 years: Institute of Medicine, Committee on Quality of Health Care in America, Crossing the Quality Chasm: A New Health System for the 21st Century, National Academy Press, Washington, D.C. (2001).
One funeral at a time: Various formulations of this quotation have been attributed to Max Planck, Albert Einstein, and Paul Samuelson.
Few doctors rely on research to treat individual patients: Brandi White, “Making Evidence-Based Medicine Doable in Everyday Practice,” Family Practice Mgt. (Feb. 2004), http://www.aafp.org/fpm/20040200/51maki.htm; D. M. Windish and M. Diener-West, “A Clinician-Educator’s Roadmap to Choosing and Interpreting Statistical Tests,” 21 J. Gen. Intern. Med. 656 (2006); Kevin Patterson, “What Doctors Don’t Know (Almost Everything),” N.Y. Times Magazine, May 5, 2002.
Where is the library in the doctor’s office?: A law library is physically and intellectually at the heart of many law practices. A client comes to a lawyer for advice, and the lawyer hits the books to find out what the law is for that client’s particular problem. Until evidence-based medicine came along, most doctors didn’t routinely research the problems of individual patients. The physician might try to keep up in the field generally. He or she might subscribe to the New England Journal of Medicine and JAMA. Yet rare would be the case where a physician would pick up a book or journal article to do patient-specific research. In sharp contrast to a law office, most doctors’ offices never had a library. If a physician didn’t know the answer, she might consult with a specialist, but neither the physician nor the specialist was very likely to pick up a book and read.
Information must be retrievable to be useful: Gordon Guyatt et al., “Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine,” 268 JAMA 2420 (1992); “Center for Health Evidence, Evidence-Based Medicine: A New Approach to Teaching the Practice of Medicine” (2001), http://www.cche.net/usersguides/ebm.asp; Lisa Sanders, “Medicine’s Progress, One Setback at a Time,” N.Y. Times, Mar. 16, 2003, p. 29; InfoPOEMs, https://www.infopoems.com.
Data-based diagnostic decisions: “To the Editor,” N. Engl. J. Med., Nov. 10, 2005.
Diagnostic-decision support software: The major softwares in this field include Isabel, QMR, Iliad, Dxplain, DiagnosisPro, and PKC.
Super Crunching away diagnostic errors: Jeanette Borzo, “Software for Symptoms,” Wall St. J. (Office Technology), May 23, 2005, p. R10; Jason Maude biography, The Beacon Charitable Trust, http://www.beaconfellowship.org.uk/ biography2003_11.asp; David Leonhardt, “Why Doctors So Often Get It Wrong,” N.Y. Times, Feb. 22, 2006.
Isabel helps doctors consider major diagnosis: Stephen M. Borowitz, et al., “Impact of a Web-based Diagnosis Reminder System on Errors of Diagnosis,” presented at AMIA 2006: Biomedical and Health Informatics (Nov. 11, 2006).
CHAPTER 5
Super Crunchers take on experts to predict Supreme Court decisions: Andrew D. Martin et al., “Competing Approaches to Predicting Supreme Court Decision Making,” 2 Persp. on Pol., 763 (2004); Theodore W. Ruger et al., “The Supreme Court Forecasting Project: Legal and Political Science Approaches to Predicting Supreme Court Decision-making,” 104 Colum. L. Rev. 1150 (2004).
Holmes on legal positivism: Oliver W. Holmes, The Common Law 1 (1881) (“The prophesies of what the courts will do in fact, and nothing more pretentious, are what I mean by the law.”). See also Oliver Wendell Holmes, Jr., “The Path of the Law,” 10 Harv. L. Rev. 457, 461 (1897) (“The object of our study, then, is prediction, the prediction of the incidence of the public force through the instrumentality of the courts.”).
Langdell on law as a science: Christopher C. Langdell, “Harvard Celebration Speeches,” 3 L.Q. Rev. 123, 124 (1887).
Mee
hl’s little book (and other works): Paul E. Meehl, Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence (1954). See also William M. Grove, “Clinical Versus Statistical Prediction: The Contribution of Paul E. Meehl,” 61 J. Clinical Psychol. 1233 (2005), http://www.psych. umn.edu/faculty/grove/112clinicalversusstatisticalprediction.pdf; Michael P. Wittman, “A Scale for Measuring Prognosis in Schizophrenic Patients,” 4 Elgin Papers 20 (1941); Drew Western and Joel Weinberger, “In Praise of Clinical Judgment: Meehl’s Forgotten Legacy,” 61 J. Clinical Psychol. 1257, 1259 (2005), http://www.psychsystems.net/lab/2005_w_weinberger_meeh_JCP.pdf; Paul E. Meehl, in 8 A History of Psychology in Autobiography 337, 354 (G. Lindzey, ed., 1989).
Snijders vs. the buying experts: Chris Snijders et al., “Electronic Decision Support for Procurement Management: Evidence on Whether Computers Can Make Better Procurement Decisions,” 9 J. Purchasing and Supply Mgmt 191(2003); Douglas Heingartner, “Maybe We Should Leave That Up to the Computer,” N.Y. Times, Jul. 18, 2006.
Man vs. machine meta analysis: William M. Grove and Paul E. Meehl, “Comparative Efficiency of Informal (Subjective, Impressionistic) and Formal (Mechanical, Algorithmic) Prediction Procedures: The Clinical–Statistical Controversy,” 2 Psychol. Pub. Pol’y and L. 293, 298 (1996); William M. Grove, “Clinical Versus Statistical Prediction: The Contribution of Paul E. Meehl,” 61 J. Clinical Psychol. 1233 (2005), http://www.psych.umn.edu/faculty/grove/112 clinicalversusstatisticalprediction.pdf.
Human bias: D. Kahneman et al., Judgment Under Uncertainty: Heuristics and Biases (1982); R. M. Dawes and M. Mulford, “The False Consensus Effect and Overconfidence: Flaws in Judgment, or Flaws in How We Study Judgment?” 65 Organizational Behavior and Human Decision Processes 201 (1996).
A pool is more dangerous than a gun: Steven Levitt, editorial, “Pools More Dangerous Than Guns,” Chi. Sun-Times, Jul. 28, 2001, p. 15.