The Signal and the Noise

by Nate Silver

  27. The most junior weather forecasters might start out at grade 5 of the government’s general pay scale, where salaries are about $27,000 per year before cost-of-living adjustments. The top salary for a government employee under the schedule is about $130,000 plus cost-of-living adjustments.

  28. “National Weather Service: FY 2012 Budget Highlights,” National Oceanic and Atmospheric Administration, United States Department of Commerce. http://www.corporateservices.noaa.gov/nbo/FY09_Rollout_Materials/NWS_One_Pager_FINAL.pdf.

  29. “Weather Impact on USA Economy,” National Oceanic and Atmospheric Association Magazine, Nov. 1, 2001. http://www.magazine.noaa.gov/stories/mag4.htm.

  30. This is probably not entirely coincidental—weather forecasting is a 24/7 business, and everyone at the World Weather Building takes their rotation on the night shift. With the lack of sunlight and windows at the World Weather Building, I sometimes had the feeling of being in a submarine.

  31. “HPC% Improvement to NCEP Models (1-Inch Day 1 QPF Forecast),” Hydro Meteorological Prediction Center, National Oceanic and Atmospheric Association. http://www.hpc.ncep.noaa.gov/images/hpcvrf/1inQPFImpann.gif.

  32. “HPC Pct Improvement vs MOS (Max Temp MAE: Stations Adjusted >= 1 F),” Hydro Meteorological Prediction Center, National Oceanic and Atmospheric Association. http://www.hpc.ncep.noaa.gov/images/hpcvrf/max1.gif.

  33. “Weather Fatalities,” National Weather Service, National Oceanic and Atmospheric Association. http://www.nws.noaa.gov/om/hazstats/images/weather_fatalities.pdf.

  34. “NHC Tropical Cyclone Forecast Verification,” National Hurricane Center, National Weather Service, National Oceanic and Atmospheric Association; updated March 1, 2012. http://www.nhc.noaa.gov/verification/verify5.shtml.

  35. Another type of competition is for the taxpayer’s dollar. So long as the memory of Hurricane Katrina lingers—an event which, in addition to its human toll, established the precedent that the government is held accountable for its response to natural disasters—the Weather Service is probably immune from massive budget cuts. But concerns over budgets remain a constant paranoia in Camp Springs—the fear is that some bright bulb in Washington will get wind of how well the computers are doing, and decide that the human forecasters are redundant. President Obama’s proposed 2013 budget for the Weather Service increased funding for weather satellites but cut it for basic operations and research.

  36. Traffic estimates are per Alexa.com.

  37. Although ten inches of snow sounds like a lot more precipitation than one inch of rain, they’re in fact roughly equivalent, because snow molecules are much less dense. If ten inches of snow were to melt, they would typically leave about one inch of water.

  38. Allan H. Murphy, “What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting,” American Meteorological Society 8 (June 1993): pp. 281–293. http://www.swpc.noaa.gov/forecast_verification/Assets/Bibliography/i1520-0434-008-02-0281.pdf.

  39. “History for Lansing, MI: Friday January 13, 1978,” Wunderground.com. http://www.wunderground.com/history/airport/KLAN/1978/1/13/DailyHistory.html?req_city=Lansing&req_state=MI&req_statename=Michigan.

  40. Data courtesy of Eric Floehr, ForecastWatch.com.

  41. In fact, predicting a 50 percent chance of rain is a fairly bold forecast, since it typically rains on only about 20 percent of the days at an average location in the United States.

  42. At one point in advance of the 2012 presidential election, I told a group of executives I had been invited to speak with that I thought the best forecast at the time was that Barack Obama and Mitt Romney each had a 50 percent chance of winning. They demanded that I stop being so wishy-washy and give them a real answer.

  43. The wet bias that Floehr identified refers to predicting precipitation more often than it occurs in reality. It does not necessarily imply that the forecasters predict too little precipitation conditional on rain occurring. In fact, Floehr has also found that the weather forecasters tend to underestimate precipitation associated with the most severe winter storms, storms like the “Snowpocalypse” in New York City in 2012.

  44. Although calibration is a very important way to judge a forecast, it won’t tell you everything. Over the long run, for instance, it rains about 20 percent of the days at a randomly chosen location in the United States. So you can have a well-calibrated forecast just by guessing that there is always a 20 percent chance of rain. However, this forecast has no real skill; you are just defaulting to climatology. The counterpart to calibration is what is called discrimination or resolution, which is a measure of how much you vary your forecasts from one case to the next. A forecaster who often predicts a 0 percent chance of rain, or a 100 percent chance, will score better on discrimination than one who always guesses somewhere in the middle.

  Good evaluations of forecasts account for both of these properties—either individually or with statistical measures like the Brier score that attempt to account for both properties at once.

  The reason I say calibration is the best measure of a forecast is pragmatic: most expert forecasters have no trouble with discrimination. In fact, they discriminate too much—that is, their forecasts are overconfident.

  45. “Performance Characteristics and Biases of the Operational Forecast Models,” National Weather Service Weather Forecast Office, Louisville, KY; National Weather Service, National Oceanic and Atmospheric Association; May 23, 2004. http://www.crh.noaa.gov/lmk/soo/docu/models.php.

  46. Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D. Phillips, “Calibration of Probabilities: The State of the Art to 1980,” Decision Research, Perceptronics, for Office of Naval Research, 1986. http://www.dtic.mil/cgi-bin/GetTRDoc?AD=ADA101986.

  47. J. Eric Bickel and Seong Dae Kim, “Verification of the Weather Channel Probability of Precipitation Forecasts,” American Meteorological Society 136 (December 2008): pp. 4867–4881. http://faculty.engr.utexas.edu/bickel/Papers/TWC_Calibration.pdf.

  48. J. D. Eggleston, “How Valid Are TV Weather Forecasts?” Freakonomics.com, Apr. 21, 2008. http://www.freakonomics.com/2008/04/21/how-valid-are-tv-weather-forecasts/comment-page-6/#comments.

  49. Per interview with Max Mayfield.

  50. Mayfield was born September 19, 1948, and was fifty-six when Katrina hit; he is in his midsixties now.

  51. The cone of uncertainty is officially supposed to cover two-thirds of the potential landfall locations, although according to Max Mayfield, storms have remained within the cone somewhat more often than that in practice.

  52. “Vermont Devastation Widespread, 3 Confirmed Dead, 1 Man Missing,” BurlingtonFreePress.com, August 29, 2011. http://www.burlingtonfreepress.com/article/20110829/NEWS02/110829004/Vermont-devastation-widespread-3-confirmed-dead-1-man-missing.

  53. Associated Press, “Hurricane Rita Bus Owner Found Guilty,” USA Today, October 3, 2006. http://www.usatoday.com/news/nation/2006-10-03-rita-bus_x.htm.

  54. “Hurricane Katrina Timeline,” The Brookings Institution. http://www.brookings.edu/fp/projects/homeland/katrinatimeline.pdf.

  55. Douglas Brinkley, “How New Orleans Drowned,” Vanity Fair, June 2006. http://www.vanityfair.com/politics/features/2006/06/brinkley_excerpt200606.

  56. Keith Elder, et al., “African Americans’ Decisions Not to Evacuate New Orleans Before Hurricane Katrina: A Qualitative Study,” American Journal of Public Health 97, supplement 1 (April 2007). http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1854973/.

  57. H. Gladwin and W. G. Peacock, “Warning and Evacuation: A Night for Hard Houses,” in Hurricane Andrew: Ethnicity, Gender, and the Sociology of Disasters (Oxford, England: Routledge, 1997), pp. 52–74.

  58. Brinkley, “How New Orleans Drowned.”

  59. “Hurricane Katrina Timeline,” Brookings Institution.

  60. “Houston Shelter Residents’ Reports of Evacuation Orders and Their Own Evacuation Experiences,” in “Experiences of Hurricane Katrina Evacuees in Houston Shelters: Implications for Future Planning,” by Moll
yann Brodie, Erin Weltzien, Drew Altman, Robert J. Blendon and John M. Benson, American Journal of Public Health 9, no. 8 (August2006): pp. 1402–1408. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1522113/table/t2/.

  61. Amanda Ripley, The Unthinkable (New York: Random House, 2008). Kindle edition.

  CHAPTER 5: DESPERATELY SEEKING SIGNAL

  1. John Dollar, “The Man Who Predicted the Earthquake,” Guardian, April 5, 2010. http://www.guardian.co.uk/world/2010/apr/05/laquila-earthquake-prediction-giampaolo-giuliani.

  2. “Scientists in the Dock,” Economist, September 17, 2011. http://www.economist.com/node/21529006.

  3. Roger A. Pielke Jr., “Lessons of the L’Aquila Lawsuit,” Bridges 31 (October 2011). http://sciencepolicy.colorado.edu/admin/publication_files/2011.36.pdf.

  4. “Eyewitnesses: Italy Earthquake,” BBC News, April 6, 2009. http://news.bbc.co.uk/2/hi/europe/7985248.stm.

  5. Michael Taylor, “L’Aquila, Guiliani, and the Price of Earthquake Prediction,” The Pattern Connection, July 7, 2010. http://patternizer.wordpress.com/2010/07/07/laquila-guiliani-and-the-price-of-earthquake-prediction/.

  6. John Bingham, “L’Aquila Dogged by Earthquakes Through 800 Year History,” Telegraph, April 6, 2009. http://www.telegraph.co.uk/news/worldnews/europe/italy/5113215/LAquila-dogged-by-earthquakes-through-800-year-history.html.

  7. M. Stucchi, C. Meletti, A. Ravida, V. D’Amio, and A. Capera, “Historical Earthquakes and Seismic Hazard of the L’Aquila Area,” Progettazione Sismica 1, no. 3 (2010): pp. 23–24.

  8. Elisabeth Malkin, “Once Built on a Lake, Mexico City Now Runs Dry,” New York Times, March 16, 2006. http://www.nytimes.com/2006/03/16/world/americas/16iht-mexico.html.

  9. Nicola Nosengo, “Italian Earthquake Toll Highlights Poor Preparedness,” Nature news blog, May 22, 2012. http://blogs.nature.com/news/2012/05/italian-earthquake-toll-highlights-poor-preparedness.html.

  10. “Così Posso Prevedere I Terremoti In Abruzzo Ci Sono 5 Apparecchi,” La Repubblica, April 6, 2009. Translated into English using Google Translate. http://www.repubblica.it/2009/04/sezioni/cronaca/terremoto-nord-roma/giulianigiampaolo/giulianigiampaolo.html.

  11. Symon Hill, “Earthquakes and Bad Theology,” Symon Hill’s Blog, Ekklesia, January 17, 2010. http://www.ekklesia.co.uk/node/11032.

  12. William Pike, “The Haiti and Lisbon Earthquakes: ‘Why, God?’ ” Encyclopedia Britannica blog, January 19, 2010. http://www.britannica.com/blogs/2010/01/the-haiti-and-lisbon-earthquakes-why-god/.

  13. Rick Brainard, “The 1755 Lisbon Earthquake,” 18th Century History, 2005. http://www.history1700s.com/articles/article1072.shtml.

  14. Susan Hough, “Confusing Patterns with Coincidences,” New York Times, April 11, 2009. http://www.nytimes.com/2009/04/12/opinion/12hough.html.

  15. John Roach, “Can the Moon Cause Earthquakes?” National Geographic News, May 23, 2005. http://news.nationalgeographic.com/news/2005/05/0523_050523_moonquake.html.

  16. Since 1900, the ten deadliest earthquakes worldwide have killed about 1.6 million people, versus 1.2 million for hurricanes. Matt Rosenberg, “Top 10 Deadliest World Hurricanes Since 1900,” About.com. http://geography.about.com/od/physicalgeography/a/deadlyhurricane.htm; “Earthquakes with 1,000 or More Deaths Since 1900,” United States Geological Service. http://earthquake.usgs.gov/earthquakes/world/world_deaths.php.

  17. I speak here of the perception, not of the reality. In a literal sense, earthquakes are much more common than hurricanes, with several million occurring worldwide every year compared with only several dozen hurricanes. However, the vast majority of these are tiny and are undetectable unless you have a seismometer, whereas hurricanes almost always make news.

  18. “Legends of Unusual Phenomena Before Earthquakes—Wisdom or Superstition?” in Earthquakes and Animals—From Folk Legends to Science (Hackensack, NJ: World Scientific Publishing, 2005). www.worldscibooks.com/etextbook/5382/5382_chap01.pdf.

  19. Giuliani was given a very favorable write-up in the estimable UK newspaper the Guardian, for instance. Dollar, “The Man Who Predicted the Earthquake.”

  20. Ed Wilson, Don Drysdale, and Carrie Reinsimar, “CEPEC Keeps Eye on Earthquake Predictions,” State of California Department of Conservation, October 23, 2009. http://www.consrv.ca.gov/index/news/Pages/CEPECKeepsEyeOnEarthquakePredictions.aspx.

  21. R. A. Grant and T. Halliday, “Predicting the Unpredictable: Evidence of Pre-Seismic Anticipatory Behaviour in the Common Toad,” Journal of Zoology 700 (January 25, 2010). http://image.guardian.co.uk/sys-files/Environment/documents/2010/03/30/toads.pdf.

  22. One obvious problem with the paper: the timing of the toads’ behavior coincides very well with the foreshocks to L’Aquila—much better than with the main quake. So if they were sensitive to earthquakes, it seems more likely that they were reacting to these foreshocks, rather than “predicting” the main earthquake. Of course, seismometers were able to detect the foreshocks as well.

  23. Per Bak, How Nature Works: The Science of Self-Organized Criticality (New York: Springer, 1999). Kindle edition, location 1357.

  24. “FAQs—Earthquake Myths,” United States Geological Survey. http://earthquake.usgs.gov/learn/faq/?categoryID=6&faqID=13.

  25. According to the USGS, there is a 95.4 percent chance of an earthquake of at least magnitude 6.75 hitting somewhere in the 100-kilometer radius surrounding San Francisco within the next 100 years, and a 97.6 percent chance for Los Angeles. The probability of at least one of the cities experiencing a major earthquake is 99.9 percent.

  26. “2009 Earthquake Probability Mapping,” United States Geological Survey. http://geohazards.usgs.gov/eqprob/2009/.

  27. “Earthquake Facts and Statistics,” United States Geological Survey. http://earthquake.usgs.gov/earthquakes/eqarchives/year/eqstats.php.

  28. The exceptions are in wealthy and earthquake-rich regions like California, Japan, and Italy, where there are seismometers on every proverbial block.

  29. 1964 marks about the point at which record keeping for medium-size earthquakes significantly improved.

  30. “Composite Earthquake Catalog,” Advanced National Seismic System, Northern California Earthquake Data Center. http://quake.geo.berkeley.edu/cnss/.

  31. In a box measuring three degrees latitude by three degrees longitude centered around Tehran.

  32. “Largest and Deadliest Earthquakes by Year: 1990–2011,” United States Geological Survey. http://earthquake.usgs.gov/earthquakes/eqarchives/year/byyear.php.

  33. “Corruption Perceptions Index 2011,” Transparency.org. http://cpi.transparency.org/cpi2011/.

  34. Kishor Jaiswal and David Wald, “An Empirical Model for Global Earthquake Fatality Estimation,” Earthquake Spectra 26, no. 4 (November 2010). http://earthquake.usgs.gov/earthquakes/pager/prodandref/Jaiswal_&_Wald_(2010)_Empirical_Fatality_Model.pdf.

  35. An empirically derived relationship known as Omori’s Law dictates that the number of aftershocks is inversely proportional to the amount of time that has passed since the initial earthquake hit. In other words, aftershocks are more likely to occur immediately after an earthquake than days later, and more likely to occur days later than weeks after the fact.

  36. This chart and the others cover a grid of 1 degree of latitude by 1 degree of longitude, with the epicenter of the main quake at the center of the box. The exception is figure 5-4d (for Reno, Nevada) since there was no main earthquake; Reno’s city hall serves as the center of the box in that case.

  37. Instituto Nazionale di Geofisica e Volcuanologia database. http://web.archive.org/web/20101114020542/http://cnt.rm.ingv.it/earthquakes_list.php.

  38. The exact magnitude of the earthquake is debated; the database I used to generate the charts list it at magnitude 9.1, while other sources have it at 9.0.

  39. Hough, “Predicting the Unpredictable,” Kindle locations 1098–1099.

  40. Ibid., Kindle locations 1596–1598.

  41. Ibid., Kindle locations 1635–1636.

  42. Anchorage Daily
News, June 27, 1981, p. A-10.

  43. Hough, “Predicting the Unpredictable,” Kindle location 1706.

  44. W. H. Bakun and A. G. Lindh, “The Parkfield, California, Earthquake Prediction Experiment,” Science 229, no. 4714 (August 16, 1985). http://earthquake.usgs.gov/research/parkfield/bakunLindh85.html.

  45. Hough, “Predicting the Unpredictable,” Kindle locations 488–491.

  46. Per interview with John Rundle.

  47. “UCLA Geophysicist Warns 6.4 Quake to Hit LA by Sept. 5,” Agence France-Presse via SpaceDaily.com, April 15, 2004. http://www.spacedaily.com/news/tectonics-04d.html.

  48. P. Shebalin, V. Keilis-Borok, A. Gabrielov, I. Zaliapin, and D. Turcotte, “Short-term Earthquake Prediction by Reverse Analysis of Lithosphere Dynamics,” Tectonophysics 413 (December 13, 2005). http://www.math.purdue.edu/~agabriel/RTP_Tect.pdf.

  49. J. Douglas Zechar and Jiancang Zhuang, “Risk and Return: Evaluating Reverse Tracing of Precursors Earthquake Predictions,” Geophysical Journal International (May 17, 2010). http://bemlar.ism.ac.jp/zhuang/pubs/zechar10.pdf.

  50. Likewise, a Google News search returns no record of Keilis-Borok’s predictions in 2003.

  51. My sense from speaking with Dr. Keilis-Borok is that he was acting in good faith, but forecasters do not always do so. Sometimes, especially in economic forecasting, they’ll even blatantly cheat by revising their method after the fact and claiming the new model would have predicted an event that they missed in real time. There is simply no substitute for having a clear public record of a prediction made before an event occurred.

  52. Andrew Bridges, “Decade After Northridge, Earthquake Predictions Remain Elusive,” Associated Press State & Local Wire, January 12, 2004.

  53. “Information About the Keilis-Borok California Earthquake Prediction,” United States Geological Survey. http://earthquake.usgs.gov/earthquakes/eqinthenews/2004/KB_prediction.php.

  54. Zechar and Zhuang, “Risk and Return: Evaluating Reverse Tracing of Precursors Earthquake Predictions.”

  55. Arnaud Mignan, Geoffrey King, and David Bowman, “A Mathematical Formulation of Accelerating Moment Release Based on the Stress Accumulation Model,” Journal of Geophysical Research 112, BO7308 (July 10, 2007). http://geology.fullerton.edu/dbowman/Site/Publications_files/Mignan_etal_JGR2007_1.pdf.