Black Box Thinking

by Matthew Syed

  28. http://www.ncbi.nlm.nih.gov/pubmed/21471476.

  29. http://www.nytimes.com/2002/10/15/us/alphonse-chapanis-dies-at-85-was-a-founder-of-ergonomics.html.

  30. Kim Phong L. Vu and Robert Proctor, Handbook of Human Factors in Web Design, 2nd ed. (New York: CRC Press, 2004).

  CHAPTER 2: UNITED AIRLINES 173

  1. Information relating to the flight sourced from the National Transportation Safety Board Aircraft Accident Report (December 28, 1978); “Focused on Failure,” episode from the Mayday TV series; various news sources; and multiple interviews with investigators, pilots, and investigators.

  2. http://www.eurohoc.org/task/task_docs/CAPAP2002_02.pdf.

  3. http://www.airdisaster.com/reports/ntsb/AAR78-08.pdf.

  4. http://www.airdisaster.com/reports/ntsb/AAR73-14.pdf.

  5. Malcolm Gladwell, Outliers: The Story of Success (New York: Penguin, 2009).

  6. This also hints at why so many scientific discoveries are made “accidentally,” such as penicillin, etc.

  7. J. Vanden Bos et al., “The $17.1 Billion Problem: The Annual Cost of Measurable Medical Errors,” Health Affairs 30, no. 4 (2011): 596–603.

  8. Sidney Dekker of Griffith University has lectured extensively about how the model used to understand why accidents happen has evolved over the last century and a half, in many remarkable and fascinating ways.

  9. Oskar Morgenstern, Abraham Wald obituary, Econometrica, October 1951.

  10. H. Freeman, “Abraham Wald,” in D. L. Sills (ed.), International Encyclopedia of Social Sciences, 16 (1968), 435–38.

  11. Karl Menger, “The Formative Years of Abraham Wald and His Work in Geometry,” in Annals of Mathematical Statistics 23, no. 1 (1952): 14–20.

  12. http://youarenotsosmart.com/tag/abraham-wald/.

  13. Ibid.

  14. Ibid.

  15. https://hbr.org/2011/04/strategies-for-learning-from-failure.

  16. http://cna.org/sites/default/files/research/0204320000.pdf.

  17. Oskar Morgenstern, Abraham Wald obituary.

  CHAPTER 3: THE PARADOX OF SUCCESS

  1. Information on United 1549 from investigation report (http://www.ntsb.gov/investigations/AccidentReports/Reports/AAR1003.pdf), two Mayday National Geographic documentaries on the flight, and various media reports.

  2. http://content.time.com/time/specials/packages/article/0,28804,1894410_1894289_1894258,00.html.

  3. Karl Popper, from Conjectures and Refutations: The Growth of Scientific Knowledge (London: Routledge & Kegan Paul, 1963).

  4. The story is a little more complex. The experiment only partially confirmed Einstein’s theory. It took further experiments for scientists to universally agree that light is attracted to heavy bodies.

  5. See Karl Popper, Conjectures and Refutations.

  6. Philip H. Gosse, Omphalos: An Attempt to Untie the Geological Knot (Rochester, NY: Scholar’s Choice, 2015).

  7. Karl Popper, Conjectures and Refutations.

  8. This example is cited in Bryan Magee’s Philosophy and the Real World: An Introduction to Karl Popper (Chicago: Open Court Publishing, 1985).

  9. Nassim N. Taleb, The Black Swan: The Impact of the Highly Improbable (New York: Penguin, 2008).

  10. Daniel Kahneman and Gary Klein, “Conditions for Intuitive Expertise, Failure to Disagree,” American Psychologist 64, no. 6 (2009): 515–26.

  11. Ibid.

  12. K. Anders Ericsson (ed.), Development of Professional Expertise: Toward Measurement of Expert Performance and Design of Optimal Learning Environments (New York: Cambridge University Press, 2009).

  13. Such a system has been recommended by K. Anders Ericsson in Development of Professional Expertise: Toward Measurement of Expert Performance and Design of Optimal Learning Environments (New York: Cambridge University Press, 2009). A similar system driven by objective feedback has been proposed for psychotherapists by Terence Tracey of Arizona State University and colleagues. See Tracey et al., “Expertise in Psychotherapy: An Elusive Goal?,” American Psychologist 69, no. 3 (2014): 218–29.

  14. http://www.telegraph.co.uk/news/nhs/10940874/Can-the-Japanese-car-factory-methods-that-transformed-a-Seattle-hospital-work-on-the-NHS.html?mobile=basic.

  15. Ibid.

  16. Ibid.

  17. Charles Kenney, Transforming Health Care: Virginia Mason Medical Center’s Pursuit of the Perfect Patient Experience (London: Productivity Press, 2010).

  18. http://www.nytimes.com/2008/05/18/us/18apology.html?pagewanted=all.

  19. http://www.nytimes.com/2008/05/18/us/18apology.html?pagewanted=print&_r=0.

  20. Peter Pronovost, lecture on System Safety at Johns Hopkins University.

  21. http://www.nytimes.com/2010/03/09/science/09conv.html.

  22. Peter Pronovost, Safe Patients, Smart Hospitals: How One Doctor’s Checklist Can Help Us Change Health Care from the Inside Out (New York: Plume,2004).

  23. Atul Gawande, Complications.

  24. The Francis Report: http://www.midstaffspublicinquiry.com/sites/default/files/report/Executive%20summary.pdf.

  25. The Kirkup Report: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/408480/47487_MBI_Accessible_v0.1.pdf.

  26. Select Committee Report: http://www.publications.parliament.uk/pa/cm201415/cmselect/cmpubadm/886/88602.htm.

  27. http://www.deadbymistake.com/.

  28. Michael Gillam et al., “The Health Care Singularity and the Age of Semantic Medicine” in The Fourth Paradigm: Data-Intensive Scientific Discovery (Microsoft, 2009).

  29. Atul Gawande, The Checklist Manifesto: How to Get Things Right (London: Profile, 2010).

  30. Atul Gawande, Complications.

  31. http://www.nytimes.com/2009/05/26/health/26autopsy.html?pagewanted=all&_r=0.

  32. Atul Gawande, Complications.

  33. http://www.pbs.org/wgbh/pages/frontline/criminal-justice/post-mortem/more-deaths-go-unchecked-as-autopsy-rate-falls-to-miserably-low-levels/.

  34. James Reason, A Life in Error: From Little Slips to Big Disasters (Burlington, VT: Ashgate, 2013).

  35. http://www.chfg.org/resources/07_qrt04/Anonymous_Report_Verdict_and_Corrected_Timeline_Oct_07.pdf.

  CHAPTER 4: WRONGFUL CONVICTIONS

  1. William Blackstone, Commentaries on the Laws of England (London: Forgotten Books, 2012).

  2. Edwin M. Borchard, Convicting the Innocent and State Indemnity for Errors of Criminal Justice (Seattle: Justice Institute, 2013).

  3. Ibid.

  4. Carole McCartney, “Building Institutions to Address Miscarriages of Justice in England and Wales: ‘Mission Accomplished?,’” University of Cincinnati Law Review 80, no. 4 (2013).

  5. For more on DNA in criminal justice see Jim Dwyer, Barry Scheck, and Peter Neufeld, Actual Innocence: When Justice Goes Wrong and How to Make It Right (New York: New American Library, 2003); and David Lazer et al., DNA and the Criminal Justice System: The Technology of Justice (Cambridge, MA: MIT Press, 2004).

  6. http://www.nhs.uk/Conditions/Blood-groups/Pages/Introductions.aspx.

  7. Quote taken from Jim Dwyer, Barry Scheck and Peter Neufeld, Actual Innocence.

  8. For an analysis of some of the pitfalls of interpreting very small fragments of DNA see David Bentley QC, “DNA and Case Preparation,” Law Society Gazette, January 2015.

  9. Information on wrongful convictions from the Innocence Project, state databases, and various interviews.

  10. Information on Michael Shirley taken from court documents, reports, and two author interviews with his attorney, Anita Bromley.

  11. Innocence Project.

  12. Samuel R. Gross et al., “Exonerations in the United States, 1989 through 2003,” Journal of Criminal Law and Criminology 95, no. 2 (2005): 523–60.

  13. Dwyer, Scheck, and Neufeld, Actual Innocence.r />
  14. Leon Festinger, Henry Riecken, and Stanley Schachter, When Prophecy Fails (Eastford, CT: Martino Fine Books, 2009).

  15. Poll cited in Carol Tavris and Elliot Aronson, Mistakes Were Made (but Not by Me): Why We Justify Foolish Beliefs, Bad Decisions and Hurtful Acts (London: Pinter & Martin, 2013).

  16. Elliot Aronson and Judson Mills, “The Effect of Severity of Initiation on Liking for a Group,” Journal of Abnormal and Social Psychology 59 (September 1959).

  17. Carol Tavris and Elliot Aronson, Mistakes Were Made (but Not by Me).

  18. Charles Lord, Lee Ross, and Mark Lepper, “Biased Assimilation and Attitude Polarisation: The Effects of Poor Theories on Subsequently Considered Evidence,” Journal of Personality and Social Psychology 37 (1979).

  19. Innocence Project website.

  20. Kathryn Schulz, Being Wrong: Adventures in the Margin of Error (London: Portobello Books, 2011).

  21. Quoted in Tavris and Aronson, Mistakes Were Made (but Not by Me).

  22. Schulz, Being Wrong.

  23. http://www.nytimes.com/2011/11/27/magazine/dna-evidence-lake-county.html?_r=0.

  24. Ibid.

  25. Dwyer, Scheck, and Neufeld, Actual Innocence.

  26. Ibid.

  27. Innocence Project.

  28. Dwyer, Scheck, and Neufeld, Actual Innocence.

  CHAPTER 5: INTELLECTUAL CONTORTIONS

  1. Tavris and Aronson, Mistakes Were Made (but Not by Me).

  2. John Banja, Medical Errors and Medical Narcissism (Burlington, MA: Jones & Bartlett, 2005).

  3. Ibid.

  4. Ibid.

  5. Jeff Stone and Nicholas C. Fernandez, “How Behaviour Shapes Attitudes: Cognitive Dissonance Processes,” in William D. Crano and Radmila Prislin (ed.), Attitudes and Attitude Change (New York: Psychology Press, 2013).

  6. http://www.tonyblairoffice.org/news/entry/iraq-syria-and-the-middle-east-an-essay-by-tony-blair/.

  7. http://blogs.wsj.com/economics/2010/11/15/open-letter-to-ben-bernanke/.

  8. http://www.bloomberg.com/news/articles/2014-10-02/fed-critics-say-10-letter-warning-inflation-still-right.

  9. http://www.bloomberg.com/news/articles/2014-10-02/fed-critics-say-10-letter-warning-inflation-still-right.

  10. Philip E. Tetlock, Expert Political Judgement: How Good Is It? How Can We Know? (Princeton, NJ: Princeton University Press, 2006).

  11. Sydney Finkelstein, Why Smart Executives Fail: And What You Can Learn from Their Mistakes (New York: Portfolio/Penguin, 2013).

  12. Terrance Odean, “Are Investors Reluctant to Admit Their Losses?,” Journal of Finance, October 1998.

  13. Paul J. H. Schoemaker, Brilliant Mistakes: Finding Success on the Far Side of Failure (Philadelphia: Wharton Digital Press, 2011).

  14. Karl Popper, The Poverty of Historicism (London: Routledge, 2002).

  15. As recounted in Safe Patients, Smart Hospitals by Peter Pronovost.

  16. David Hilfiker, “Facing Our Mistakes,” New England Journal of Medicine 310, no. 2 (1984): 118–22.

  17. C. E. Milch et al., “Voluntary Electronic Reporting of Medical Errors and Adverse Events: An Analysis of 92,547 Reports from 26 Acute Care Hospitals,” Journal of General Internal Medicine 21, no. 2 (2006): 165–70.

  18. Peter Pronovost, Safe Patients, Smart Hospitals.

  CHAPTER 6: REFORMING CRIMINAL JUSTICE

  1. The Lysenko story is covered in dozens of articles and books, including Lysenko and the Tragedy of Soviet Science (New Brunswick, NJ: Rutgers University Press, 2006) by Valery N. Soyfer and The Lysenko Effect: The Politics of Science (Amherst, NY: Prometheus Books, 2004) by Nils Roll-Hansen.

  2. http://tauruspet.med.yale.edu/staff/edm42/IUPUI-website/emorris.tar/emorris/emorris/Ethics%20Course%2009/Journal%20articles/lysenko-nature-rev-genetics2001-nrg0901_723a.pdf.

  3. Jasper Becker, Hungry Ghosts: Mao’s Secret Famine (New York: Simon & Schuster, 1997).

  4. http://www.haydenplanetarium.org/tyson/watch/2008/06/19/george-bush-and-star-names.

  5. http://thefederalist.com/2014/09/16/another-day-another-quote-fabricated-by-neil-degrasse-tyson/.

  6. http://thefederalist.com/2014/10/02/neil-tysons-final-words-on-his-quote-fabrications-my-bad/.

  7. Giuliana Mazzoni and Amina Memon, “Imagination Can Create False Autobiographical Memories,” Psychological Science 14, no. 2 (2003): 186–88.

  8. Elizabeth F. Loftus and John C. Palmer, “Reconstruction of Automobile Destruction: An Example of the Interaction Between Language and Memory,” Journal of Verbal Learning and Verbal Behavior 13 (1974), 585–89.

  9. http://news.bbc.co.uk/1/hi/uk/4177082.stm.

  10. Innocence Project website.

  11. Gary L. Wells, Nancy K. Steblay, and Jennifer E. Dysart, “A Test of the Simultaneous vs Sequential Lineup Methods: An Initial Report of the AJS National Eyewitness Identification Field Studies,” American Judicature Society. It is worth noting that the cost of implementation of these reforms is relatively small.

  12. Innocence Project website.

  13. http://articles.chicagotribune.com/2002-09-29/news/0209290340_1_jogger-case-jogger-attack-matias-reyes.

  14. Dwyer, Scheck, and Neufeld, Actual Innocence.

  15. http://www.miamiherald.com/incoming/article1953372.html.

  16. Committee on Identifying the Needs of the Forensic Science Community National Research Council, “Strengthening Forensic Science in the United States,” report, 2009.

  17. http://www.nytimes.com/2014/12/02/opinion/why-our-memory-fails-us.html.

  18. Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso, “Extraneous Factors in Judicial Decisions,” Proceedings of the National Academy of Sciences.

  19. http://blogs.discovermagazine.com/notrocketscience/2011/04/11/justice-is-served-but-more-so-after-lunch-how-food-breaks-sway-the-decisions-of-judges/#.VYaU80Yk-So.

  20. http://articles.chicagotribune.com/2014-06-10/news/chi-dna-links-murder-and-rape-of-holly-staker-11-to-second-murder-8-years-later-20140610_1_holly-staker-dna-evidence-dna-match.

  21. Interview with author.

  22. http://www.chicagotribune.com/news/local/breaking/ct-juan-rivera-shoes-met-20141210-story.html#page=1.

  CHAPTER 7: THE NOZZLE PARADOX

  1. See Steve Jones’s lecture: https://www.youtube.com/watch?v=for_WIKgdWg. See also Owen Barder: http://www.owen.org/blog/4018.

  2. Including Karl Popper and Friedrich Hayek.

  3. Tim Harford, Adapt: Why Success Always Starts with Failure (New York: Little, Brown, 2011).

  4. Paul Omerod, quoted in ibid.

  5. Terence Kealey, The Economic Laws of Scientific Research (New York: Palgrave Macmillan, 1996).

  6. Nassim Nicholas Taleb, Antifragile: Things That Gain from Disorder (New York: Penguin, 2013).

  7. See Jonah Lehrer, The Decisive Moment (Edinburgh: Canongate, 2007), 47.

  8. Daniel Kahneman, Thinking, Fast and Slow (New York: Penguin, 2012).

  9. http://www.bbc.co.uk/news/business-27579790.

  10. Peter Sims, Little Bets: How Breakthrough Ideas Emerge from Small Discoveries (New York: Random House Business, 2012).

  11. Ibid.

  12. Ryan Babineaux and John Krumboltz, Fail Fast, Fail Often: How Losing Can Help You Win (New York: Tarcher, 2014).

  13. David Bayles and Ted Orland, Art and Fear: Observations on the Perils (and Rewards) of Artmaking (New York: Image Continuum Press, 2001).

  14. https://www.youtube.com/watch?v=vY3OtMBCEKY.

  15. Eric Ries, The Lean Startup: How Constant Innovation Creates Radically Successful Businesses (New York: Portfolio/Penguin, 2011).

  16. Ibid.

  17. Ibid.

  18. Jim Collins and Morten T. Hansen, Great by Choice: Uncertainty, Chaos and Luck—Why Some Thrive Despite Them All (New York: Random House Bus
iness, 2011).

  19. Peter Sims, Little Bets: How Breakthrough Ideas Emerge from Small Discoveries (New York: Random House Business, 2012).

  20. http://www.bbc.co.uk/news/business-27579790.

  21. Interview with the author.

  22. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1470513/.

  23. As measured by Disability Adjusted Life Years, a conventional measurement in development.

  24. Toby Ord, “The Moral Imperative toward Cost-Effectiveness in Global Health,” Center for Global Development, March 2013.

  25. http://lesswrong.com/lw/h6c/taking_charity_seriously_toby_ord_talk_on_charity/.

  26. https://www.givingwhatwecan.org/.

  CHAPTER 8: SCARED STRAIGHT?

  1. Scared Straight!, documentary directed by Arnold Shapiro, 1978.

  2. Scared Straight!, commentary by Peter Falk.

  3. See James Finckenauer, Scared Straight: The Panacea Phenomenon Revisited (London: Waveland Pv. Inc, 1998).

  4. Peter W. Greenwood, Changing Lives: Delinquency Prevention as Crime Control Policy (Chicago: University of Chicago Press, 2007).

  5. Quoted in documentary Scared Straight!

  6. Diagrams have been amended from “Test, Learn Adapt” © Crown copyright June 2012, licensed under the terms of the Open Government License. Reproduced with permission.

  7. Ben Goldacre, Bad Science (New York: Harper Perennial, 2009).

  8. Mark Henderson, The Geek Manifesto: Why Science Matters (New York: Bantam Press, 2012).

  9. See Ben Goldacre’s excellent book Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients (New York: Faber & Faber, 2014).

  10. I am grateful to Jim Manzi for making these points so clearly in his excellent book Uncontrolled (New York: Basic Books, 2012).

  11. Jonathan Sheperd, “The Production and Management of Evidence for Public Service Reform in Evidence and Policy.”

  12. Interview with author.

  13. James Finckenauer, Scared Straight.

  14. Interview with author.

  15. Ibid.

  16. Ibid.

  17. See James Finckenauer; Scott O. Lilienfeld, “Scientifically Unsupported and Supported Interventions for Childhood Psychopathology: A Summary,” Pediatrics 115, no. 3 (2005): 761–64; and Daniel P. Mears, “Towards Rational and Evidence-based Crime Policy,” Journal of Criminal Justice 35, no. 6 (2007): 667–82.