The Signal and the Noise

by Nate Silver

  64. For instance, if some economists were consistently better than others at forecasting GDP, you would expect those who made more accurate forecasts in even-numbered years (2000, 2002, 2004) to also make more accurate forecasts in odd-numbered ones (2001, 2003, 2005). But when I took the data from the Survey of Professional Forecasters and divided it up in this way, there was very little correlation between the two. Economists who made better forecasts in even-numbered years were only about average in odd-numbered ones, and vice versa.

  65. Andy Bauer, Robert A. Eisenbeis, Daniel F. Waggoner, and Tao Zha, “Forecast Evaluation with Cross-Sectional Data: The Blue Chip Surveys,” Economic Review, Federal Reserve Bank of Atlanta, 2003. http://www.frbatlanta.org/filelegacydocs/bauer_q203.pdf.

  66. David Laster, Paul Bennett, and In Sun Geoum, “Rational Bias in Macroeconomic Forecasts,” Quarterly Journal of Economics, 114, 1 (1999), pp. 293–318. http://www.newyorkfed.org/research/staff_reports/sr21.pdf.

  67. Ibid.

  68. David Reifschneider and Peter Tulip, “Gauging the Uncertainty of the Economic Outlook from Historical Forecasting Errors,” Federal Reserve Board Financial and Economics Discussion Series #2007-60 (November 2007). http://www.federalreserve.gov/pubs/feds/2007/200760/200760abs.html.

  69. Ibid.

  70. In figure 6-6, the black lines indicating the historical average are calibrated to the mean forecast for 2012 GDP growth in the November 2011 Survey of Professional Forecasters, which was 2.5 percent.

  71. Jeremy Kahn, “The Man Who Would Have Us Bet on Terrorism—Not to Mention Discard Democracy and Cryogenically Freeze Our Heads—May Have a Point (About the Betting, We Mean),” Fortune magazine via CNN Money, September 15, 2003. http://money.cnn.com/magazines/fortune/fortune_archive/2003/09/15/349149/index.htm.

  72. Robin Hanson, Futarchy: Vote Values, but Bet Beliefs (Washington, DC: George Mason University), August 2000. http://hanson.gmu.edu/futarchy.html.

  73. Felix Salmon, “Why the Correlation Bubble Isn’t Going to Burst,” Reuters, August 19, 2011. http://blogs.reuters.com/felix-salmon/2011/08/19/why-the-correlation-bubble-isnt-going-to-burst/.

  CHAPTER 7: ROLE MODELS

  1. The narrative in the first several paragraphs is based primarily on two accounts: a New York Times Magazine article dated September 5, 1976, and a more recent Slate.com article available here: Patrick Di Justo, “The Last Great Swine Flu Epidemic,” Salon.com, April 28, 2009. http://www.salon.com/news/environment/feature/2009/04/28/1976_swine_flu.

  2. A/Victoria was named for its apparent origins in the Australian state of Victoria, where Melbourne is located; it was the dominant strain of seasonal flu in the mid-1970s.

  3. Jeffery K. Taubenberger and David M. Morens, “1918 Influenza: The Mother of All Pandemics,” Emerging Infectious Diseases, 12, 1 (January 2006). http://www.webcitation.org/5kCUlGdKu.

  4. John Barr, “The Site of Origin of the 1918 Influenza Pandemic and Its Public Health Implications,” Journal of Translational Medicine, 2, 3 (January 2004).

  5. Among many other examples, see Jane E. Brody, “Influenza Virus Continues to Keep Scientists Guessing,” New York Times, July 23, 1976. http://query.nytimes.com/mem/archive/pdf?res=F30E16FB3E5B167493C1AB178CD85F428785F9.

  6. Di Justo, “The Last Great Swine Flu Epidemic.”

  7. Harold M. Schmeck Jr., “Flu Experts Soon to Rule on Need of New Vaccine,” New York Times, March 21, 1976. http://select.nytimes.com/gst/abstract.html?res=F40711FC355E157493C3AB1788D85F428785F9.

  8. Di Justo, “The Last Great Swine Flu Epidemic.”

  9. New York Times, April 23, 1976.

  10. $180 million was the projected cost of the vaccination program itself. But Democrats—sensing Ford’s urgency and desperation—also managed to attach more than a billion dollars in unrelated social welfare spending to the bill.

  11. Although there is certainly much less flu in the summer than the winter, part of the reason for the difference is that people are not looking for the flu in the summer and those with flulike symptoms may be diagnosed with other illnesses, I was told by Dr. Alex Ozonoff of Harvard University.

  12. Boyce Rensberger, “U.S. Aide Doubts a Heavy Flu Toll,” New York Times, July 2, 1976. http://select.nytimes.com/gst/abstract.html?res=F30614F83F5B167493C0A9178CD85F428785F9.

  13. New York Times, June 9, 1976.

  14. New York Times, July 20, 1976.

  15. New York Times, June 8, 1976.

  16. These are really worth watching and look like something out of a John Waters movie. See for example “1976 Swine Flu Propaganda,” by tarot1984; YouTube.com; April 27, 2009. http://www.youtube.com/watch?v=ASibLqwVbsk.

  17. Di Justo, “The Last Great Swine Flu Epidemic.”

  18. Harold M. Schmeck, “Swine Flu Program I Halted in 9 States as 3 Die After Shots; Deaths Occur in Pittsburgh,” New York Times, October 13, 1976. http://select.nytimes.com/gst/abstract.html?res=F00910F63F5A167493C1A8178BD95F428785F9.

  19. In 1976, about 3,000 people aged sixty-five and older would have died every day in the United States.

  20. The New York Times, for instance, argued the following in an October 14 editorial:

  It is conceivable that the 14 elderly people who are reported to have died soon after receiving the vaccination died of other causes. Government officials in charge of the program claim that it is all a coincidence, and point out that old people drop dead every day. The American people have even become familiar with a new statistic: Among every 100,000 people 65 to 75 years old, there will be nine or ten deaths in every 24-hour period under most normal circumstances.

  Even using the official statistic, it is disconcerting that three elderly people in one clinic in Pittsburgh, all vaccinated within the same hour, should die within a few hours thereafter. This tragedy could occur by chance, but the fact remains that it is extremely improbable that such a group of deaths should take place in such a peculiar cluster by pure coincidence.

  Although this logic is superficially persuasive, it suffers from a common statistical fallacy. The fallacy is that, although the odds of three particular elderly people dying on the same particular day after having been vaccinated at the same particular clinic are surely fairly long, the odds that some group of three elderly people would die at some clinic on some day are much shorter.

  Assuming that about 40 percent of elderly Americans were vaccinated within the first 11 days of the program, then about 9 million people aged 65 and older would have received the vaccine in early October 1976. Assuming that there were 5,000 clinics nationwide, this would have been 164 vaccinations per clinic per day. A person aged 65 or older has about a 1-in-7,000 chance of dying on any particular day; the odds of at least three such people dying on the same day from among a group of 164 patients are indeed very long, about 480,000 to one against. However, under our assumptions, there were 55,000 opportunities for this “extremely improbable” event to occur—5,000 clinics, multiplied by 11 days. The odds of this coincidence occurring somewhere in America, therefore, were much shorter—only about 8 to 1 against.

  21. Di Justo, “The Last Great Swine Flu Epidemic.”

  22. David Evans, Simon Cauchemez, and Frederick G. Hayden, “Prepandemic Immunization for Novel Influenza Viruses: ‘Swine Flu’ Vaccine, Guillain-Barré Syndrome, and the Detection of Rare Severe Adverse Events,” Journal of Infectious Diseases, 200, no. 3 (2009), pp. 321–328. www.journals.uchicago.edu/doi/pdf/10.1086/603560.

  23. Kimberly Kindy, “Officials Are Urged to Heed Lessons of 1976 Flu Outbreak,” Washington Post, May 9, 2009. www.washingtonpost.com/wp-dyn/content/article/2009/05/08/AR2009050802050.html.

  24. New York Times, December 30, 1976.

  25. There were sporadic press accounts of other cases of the swine flu, such as one in Wisconsin, but they were never confirmed by the CDC and, at the very least, never progressed beyond a single documented victim.

  26. Peter Doshi, Figure 3 in “Trends in Recorded Influenza Mortality: United States, 1900–2004,” American Jo
urnal of Public Health, 98, no. 5, May 2008. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2374803/figure/f3/.

  27. Indeed, as Carter’s margin over Ford was only 2 points nationwide, it’s possible that fiasco cost him the election. Then again, so could a lot of things.

  28. Harold M. Schmeck, “U.S. Discloses Shortage of Swine Flu Vaccine for Children 3 to 17,” New York Times, November 16, 1978. http://select.nytimes.com/gst/abstract.html?res=F70E17F9395B167493C4A8178AD95F428785F9.

  29. New York Times, May 19, 1979.

  30. Fortunately these were very mild years for the virus; had there been a major pandemic later in the 1970s, it would have been very hard to persuade people to receive vaccinations, and there could have been thousands of preventable deaths as a result.

  31. New York Times, September 5, 1976.

  32. Much less commonly, the flu can be carried by other mammalian species like whales and horses.

  33. Indeed, because of the role that animals play in the origination of influenza, it would probably be impossible to completely eradicate it for any length of time (as we have done with some other diseases, like smallpox) unless we also eradicated it in birds and pigs; see also News Staff: “Avian Flu Research Sheds Light on Swine Flu—and Why Influenza A Can Never Be Eradicated,” Scientific Blogging, Science 2.0, May 1, 2009. http://www.science20.com/news_articles/avian_flu_research_sheds_light_swine_flu_and_why_influenza_can_never_be_eradicated.

  34. John R. Moore, “Swine Productions: A Global Perspective,” Alltech Inc., Engormix.com, accessed on May 20, 2012. http://en.engormix.com/MA-pig-industry/articles/swine-production-global-perspective_124.htm.

  35. “Food Statistics > Pork Consumption per Capita (Most Recent) by Country,” NationMaster.com, accessed May 20, 2012. http://www.nationmaster.com/graph/foo_por_con_per_cap-food-pork-consumption-per-capita.

  36. “Disease and Terror,” Newsweek, April 29, 2009. http://www.thedailybeast.com/newsweek/2009/04/30/disease-and-terror.html.

  37. Although there are alternate theories that H1N1 indeed originated in Asia; see for example Donald G. McNeil Jr., “In New Theory, Swine Flu Started in Asia, Not Mexico,” New York Times, June 23, 2009. http://www.nytimes.com/2009/06/24/health/24flu.html.

  38. Tom Blackwell, “Flu Death Toll in Mexico Could Be Lower Than First Thought,” National Post, April 29, 2009. http://web.archive.org/web/20100523224652/http://www.nationalpost.com/news/story.html?id=1547114.

  39. Jo Tuckman and Robert Booth, “Four-Year-Old Could Hold Key in Search for Source of Swine Flu Outbreak,” The Guardian, April 27, 2009. http://www.guardian.co.uk/world/2009/apr/27/swine-flu-search-outbreak-source.

  40. Keith Bradsher, “Assessing the Danger of New Flu,” New York Times, April 27, 2009. http://www.nytimes.com/2009/04/28/health/28hong.html?scp=35&sq=h1n1&st=nyt.

  41. “Tracking Swine Flu Cases Worldwide,” New York Times, April 23, 2011. http://www.nytimes.com/interactive/2009/04/27/us/20090427-flu-update-graphic.html.

  42. “Report to the President on U.S. Preparations for 2009-H1N1 Influenza, President’s Council of Advisors on Science and Technology, Executive Office of the President, August 7, 2009. http://www.whitehouse.gov/assets/documents/PCAST_H1N1_Report.pdf.

  43. Carl Bialik, “Swine Flu Count Plagued by Flawed Data,” Wall Street Journal, January 23, 2010. http://online.wsj.com/article/SB10001424052748704509704575019313343580460.html.

  44. This may have been in part because the H1N1 vaccine seemed to provide some measure of protection from the seasonal strains of the flu, which for the first time in many years produced no discernible peak in January and February as it normally does.

  45. Stephen Davies, “The Great Horse-Manure Crisis of 1894,” The Freeman, 54, no. 7, September 2004. http://www.thefreemanonline.org/columns/our-economic-past-the-great-horse-manure-crisis-of-1894/.

  46. Sir William Petty, “An Essay Concerning the Multiplication of Mankind,” 1682.

  47. Tomas Frejka, “World Population Projections: A Concise History,” Center for Policy Studies, Working Papers Number 66, March 1981. http://pdf.usaid.gov/pdf_docs/PNAAR555.pdf.

  48. Haya El Nasser, “World Population Hits 7 Billion,” USA Today, October 31, 2011.

  49. Ronald Bailey, “Seven Billion People Today—Malthusians Still Wrong (and Always Will Be),” Reason.com, October 31, 2011. http://reason.com/blog/2011/10/31/seven-billion-people-today-mal.

  50. Frejka, “World Population Projections.”

  51. “U.S. HIV and AIDS Cases Reported Through December 1999,” HIV/AIDS Surveillance Report, 11, no. 2, U.S. Department of Health and Human Services, Centers for Disease Control and Prevention. http://www.cdc.gov/hiv/topics/surveillance/resources/reports/pdf/hasr1102.pdf.

  52. James M. Hyman and E. Ann Stanley, “Using Mathematical Models to Understand the AIDS Epidemic,” Mathematical Biosciences 90, pp. 415–473, 1988. http://math.lanl.gov/~mac/papers/bio/HS88.pdf.

  53. The version I applied here was to log-transform both the year variable and the AIDS-cases variable, then calculate the exponent via regression analysis. The 95 percent confidence interval on the exponent ran from about 2.2 to 3.7 by this method, with a most likely value of about 2.9. When applied ten years into the future, those relatively modest-seeming differences turn into an exceptionally broad range of possible outcomes.

  54. Richard Carter and Kamini N. Mendis, table 4 in “Evolutionary and Historical Aspects of the Burden of Malaria,” Clinical Microbiology Reviews, 15, no. 4, pp. 564–594, October 2002.

  55. Note that RO is often given as a range in the literature; I present the midpoint of that range for ease of reading. Sources: David L. Smith, F. Ellis McKenzie, Robert W. Snow, and Simon I. Hay, malaria: “Revisiting the Basic Reproductive Number for Malaria and Its Implications for Malaria Control,” PLoS Biology, 5, no. 3, March 2007. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1802755/; ebola: G. Chowell, N. W. Hengartner, C. Castillo-Chavez, P. W. Fenimore, and J. M. Hyman, “The Basic Reproductive Number of Ebola and the Effects of Public Health Measures: The Cases of Congo and Uganda,” Journal of Theoretical Biology, 229, no. 1, pp. 119–126, July 7, 2004. math.lanl.gov/~gchowell/publications/ebolaJTB.pdf; 1918 flu: Marc Lipsitch, Christina Mills, and James Robins, “Estimates of the Basic Reproductive Number for 1918 Pandemic Influenza in the United States: Implications for Policy,” Global Health Security Initiative, 2005. www.ghsi.ca/documents/Lipsitch_et_al_Submitted%2020050916.pdf; 2009 flu and seasonal flu: Todd Neale, “2009 Swine Flu More Transmissible Than Seasonal Flu,” MedPage Today, May 11, 2009. http://www.medpagetoday.com/InfectiousDisease/SwineFlu/14154; HIV/AIDS: R. M. Anderson and R. M. May, “Population Biology of Infectious Diseases: Part I,” Nature, 280, pp. 361–367, August 2, 1979; SARS: J. Wallinga and P. Teunis, “Different Epidemic Curves for Severe Acute Respiratory Syndrome Reveal Similar Impacts of Control Measures,” American Journal of Epidemiology, 160, no. 6, pp. 509–516, 2004; others: “History and Epidemiology of Global Smallpox Eradication” in “Smallpox: Disease, Prevention, and Intervention” Centers for Disease Control and Prevention. http://www.bt.cdc.gov/agent/smallpox/training/overview/pdf/eradicationhistory.pdf.

  56. “Acquired Immunodeficiency Syndrome (AIDS) Weekly Surveillance Report,” Centers for Disease Control, December 31, 1984. http://www.cdc.gov/hiv/topics/surveillance/resources/reports/pdf/surveillance84.pdf.

  57. Gregory M. Herek and John P. Capitanio, “AIDS Stigma and Sexual Prejudice,” American Behavioral Scientist, 42, pp. 1126–1143, 1999. http://psychology.ucdavis.edu/rainbow/html/abs99_sp.pdf.

  58. Marc Lacey and Elisabeth Malkin, “First Flu Death Provides Clues to Mexico Toll,” New York Times, April 30, 2009. http://www.nytimes.com/2009/05/01/health/01oaxaca.html?scp=28&sq=h1n1&st=nyt.

  59. Jo Tuckman and Robert Booth, “Four-Year-Old Could Hold Key in Search for Source of Swine Flu Outbreak,” The Guardian, April 27, 2009. http://www.guardian.co.uk/world/2009/apr/27/swine-flu-search-outbreak-source.

  60. CNN/Time/ORC International poll of Iowa Republican voter
s, December 21–27, 2011. http://i2.cdn.turner.com/cnn/2011/images/12/28/topstate3.pdf.

  61. Selzer & Company poll of Iowa Republican voters, Des Moines Register, December 27–30, 2011. http://www.desmoinesregister.com/assets/pdf/FullTopLineResults.pdf.

  62. Marshall L. Fisher, Janice H. Hammon, Walter R. Obermeyer, and Ananth Raman, “Making Supply Meet Demand in an Uncertain World,” Harvard Business Review, May 1994. http://hbr.org/1994/05/making-supply-meet-demand-in-an-uncertain-world/ar/1.

  63. I encountered a similar case when consulting for a major Hollywood movie studio in 2009. This studio, which was very sophisticated and data-driven in many ways, had a belief that a certain weekend—let’s say it was the first weekend in October—was a particularly good time to release a big film even though it didn’t correspond to anything especially important on the calendar. What had happened is that the studio had a film that performed unexpectedly well in that time slot one year, mostly because it was just a very good movie. The studio, however, attributed some of its success to the timing of its release. Every subsequent year, then, they would save one of their better releases for this same weekend in October and then market the hell out of it. Of course, a good and well-marketed film normally will do pretty well whenever it is released. Nevertheless, this fulfilled their prediction that early October was a good time to release a film and entrenched their belief further.

  64. This calculation is based on the number of children who are classified as autistic and therefore are eligible for special education programs in public schools under the federal IDEAS Act. “Table 45. Children 3 to 21 Years Old Served Under Individuals with Disabilities Education Act, Part B, by Type of Disability: Selected Years, 1976–77 through 2008–09,” Digest of Educational Statistics, National Center for Education Statistics, Institute of Education Sciences, 2010. http://nces.ed.gov/programs/digest/d10/tables/dt10_045.asp.