The Signal and the Noise
Page 51
56. Arnaud Mignan, Geoffrey King, David Bowman, Robin Lacassin, and Renata Dmowska, “Seismic Activity in the Sumatra-Java Region Prior to the December 26, 2004 (Mw=9.0-9.3) and March 28, 2005 (Mw=8.7) Earthquakes,” Earth and Planetary Science Letters 244 (March 13, 2006). http://esag.harvard.edu/dmowska/MignanKingBoLaDm_SumatAMR_EPSL06.pdf.
57. Specifically, the fit line in figure 5-6c is generated by a technique called Lowess regression. This technique is fine for many things and does not inherently lead to overfitting, but it requires you to set a smoothness parameter that will model anything from a very tight fit to a very loose one. In this case, obviously, I’ve chosen an implausibly tight fit.
58. For instance, if you apply the overfit curve to out-of-sample data—the circles from figure 5-5—it explains only about 40 percent of the variance in them. This substantial deterioration from in-sample to out-of-sample data is characteristic of an overfit model.
59. Freeman Dyson, “Turning Points: A Meeting with Enrico Fermi,” Nature 427 (January 22, 2004). http://www.nature.com/nature/journal/v427/n6972/full/427297a.html.
60. Michael A. Babyak, “What You See May Not Be What You Get: A Brief, Nontechnical Introduction to Overfitting in Regression-Type Models,” Psychosomatic Medicine 66 (February 19, 2004). http://os1.amc.nl/mediawiki/images/Babyak_-_overfitting.pdf.
61. M. Ragheb, “Fukushima Earthquake and Tsunami Station Blackout Accident.” https://netfiles.uiuc.edu/mragheb/www/NPRE%20402%20ME%20405%20Nuclear%20Power%20Engineering/Fukushima%20Earthquake%20and%20Tsunami%20Station%20Blackout%20Accident.pdf.
62. Martin Fackler, “Tsunami Warnings, Written in Stone,” New York Times, April 20, 2011. http://www.nytimes.com/2011/04/21/world/asia/21stones.html?pagewanted=all.
63. Specifically, this represents a 1-degree latitude by 1-degree longitude box, with 38.32 degrees north and 142.37 degrees east at the center.
64. Robert J. Geller, “Shake-up Time for Japanese Seismology,” Nature 472, no. 7344 (April 28, 2011). http://kitosh.k.u-tokyo.ac.jp/uploader2/src/8.pdf.
65. Specifically, the chances of it are about 20 percent.
66. The odds of a .300 hitter going 0-for-5 are about 17 percent, assuming at-bats are independent of one another.
67. Earthsky.org staff, “Scientists Did Not Expect 9.0 Magnitude Earthquake in Japan,” FastCompany.com, March 25, 2011. http://www.fastcompany.com/1742641/scientists-did-not-expect-90-magnitude-earthquake-in-japan.
68. Seth Stein and Emile A. Okal, “The Size of the 2011 Earthquake Need Not Have Been a Surprise,” Eos Transactions American Geophysical Union 92, no. 27 (July 5, 2011): p. 227. http://www.earth.northwestern.edu/people/seth/Texts/tohoku.pdf.
69. According to the ANSS catalog, there were twelve magnitude 7 earthquakes in the area measuring 10 degrees of latitude and 10 degrees of longitude in either direction from the epicenter of the Great Sumatra Earthquake of 2004, but none measuring at a magnitude 8 or greater during this period.
70. Like that of other earthquakes, the precise magnitude of the Great Sumatra Earthquake is disputed, with various estimates putting it between 9.0 and 9.3; I use a middle estimate of 9.2 here.
71. Geller, “Shake-up Time for Japanese Seismology.”
72. SilentASMR, “2 Hours of Brown Noise (Read Description),” YouTube.com, February 25, 2012. http://ww.youtube.com/watch?v=0BfyKQaf0TU.
73. Livia Borghese, “Italian Scientists on Trial Over L’Aquila Earthquake,” CNN World, September 20, 2011. http://articles.cnn.com/2011-09-20/world/world_europe_italy-quake-trial_1_geophysics-and-vulcanology-l-aquila-seismic-activity?_s=PM:EUROPE.
74. Thomas H. Jordan and Lucile M. Jones, “Operational Earthquake Forecasting: Some Thoughts on Why and How,” Seismological Research Letters 81, 4 (July/August 2010). http://earthquake.usgs.gov/aboutus/nepec/meetings/10Nov_Pasadena/Jordan-Jones_SRL-81-4.pdf.
75. Alicia Chang, “Location a Major Factor in New Zealand Earthquake Devastation,” Washington Post, February 22, 2011. http://www.washingtonpost.com/wp-dyn/content/article/2011/02/22/AR2011022205105.html.
76. Ya-Ting Leea, Donald L. Turcottea, James R. Holliday, Michael K. Sachs, John B. Rundlea, Chien-Chih Chen, and Kristy F. Tiampoe, “Results of the Regional Earthquake Likelihood Models (RELM) Test of Earthquake Forecasts in California,” Proceedings of the National Academy of Sciences of the United States of America, September 26, 2011. http://www.pnas.org/content/early/2011/09/19/1113481108.abstract?sid=ea35f085-e352-42a8-8128-19149a05c795.
CHAPTER 6: HOW TO DROWN IN THREE FEET OF WATER
1. Christopher S. Rugaber, “Unexpected Jump in Unemployment Rate to 9.2% Stings Markets,” Denver Post, July 9, 2011. http://www.denverpost.com/business/ci_18444012.
2. Christine Hauser, “Two Jobs Reports Point to a Higher Gain in June,” New York Times, July 7, 2011. http://www.nytimes.com/2011/07/08/business/economy/data-point-to-growth-in-jobs-in-june.html.
3. Based on data from the Survey of Professional Forecasters, Federal Reserve Bank of Philadelphia. http://www.phil.frb.org/research-and-data/real-time-center/survey-of-professional-forecasters/anxious-index/.
4. Roger A. Pielke Jr., “Lessons of the L’Aquila Lawsuit,” Bridges 31 (October. 2011). http://sciencepolicy.colorado.edu/admin/publication_files/2011.36.pdf.
5. Teri Tomaszkiewicz, “Disaster Isn’t Over When Media Leave: Discovering the Meaning of Memorial Day in North Dakota,” Milwaukee Journal Sentinel, June 1, 1997.
6. Ashley Shelby, Red River Rising: The Anatomy of a Flood and the Survival of an American City (St. Paul: Borealis Books, 2004).
7. In fact, about two feet worth of sandbags were deployed at Grand Forks, meaning that its total protection was about 52 or 53 feet by the time the flood hit. But that still was not quite enough to prevent a 54-foot flood.
8. This figure can be derived from the margin of error, assuming the error is distributed normally.
9. Figure 6-1 is not drawn to scale.
10. Roger A. Pielke, Jr., “Who Decides? Forecasts and Responsibilities in the 1997 Red River Flood,” Applied Behavioral Science Review 7, no. 2 (1999). http://128.138.136.233/admin/publication_files/resource-81-1999.16.pdf.
11. Pielke, “Who Decides? Forecasts and Responsibilities in the 1997 Red River Flood.”
12. Alex Veiga, “U.S. Foreclosure Rates Double,” Associated Press, November 1, 2007. http://www.azcentral.com/realestate/articles/1101biz-foreclosures01-ON.html.
13. Jonathan Stempel, “Countrywide Financial Plunges on Bankruptcy Fears,” Reuters, August 16, 2007. http://uk.reuters.com/article/2007/08/16/countrywide-financial-idUKNOA62283620070816.
14. John B. Taylor, Getting Off Track: How Government Actions and Interventions Caused, Prolonged, and Worsened the Financial Crisis (Stanford, CA: Hoover Institution Press, Kindle edition, 2009), location 361.
15. Note, however, that the economists assigned a higher chance, about 20 percent, to a negative GDP reading in each of the four financial quarters.
16. In fact, this 1-in-500 estimate is a little generous, since it applied to any GDP reading below minus 2 percent, whereas the actual GDP figure of –3.3 percent was quite a bit lower than that. Although the economists did not explicitly quantify it, it can be inferred that they would have assigned only about a 1-in-2,000 chance to a GDP reading of –3.3 percent or worse.
17. Specifically, I’ve looked at the forecasts made each November about the GDP growth in the following year; for instance, the November 1996 forecast of GDP growth in 1997.
18. Michael P. Clements, “An Evaluation of the Survey of Professional Forecasters Probability Distribution of Expected Inflation and Output Growth,” Journal of Economic Literature, November 22, 2002. http://www.icmacentre.ac.uk/pdf/seminar/clements2.pdf.
19. Based on the binomial distribution, the chance that a well-calibrated forecast would fall outside its 90 percent prediction interval six times in eighteen years is just 0.6 percent, or about 1 chance in 150.
20. This covers all releases of the Survey of Professional Fore
casters from the fourth quarter of 1968 through the fourth quarter of 2010, excluding a few early cases where the economists were not asked to issue an annual forecast.
21. Prakash Loungani, “The Arcane Art of Predicting Recessions,” Financial Times via International Monetary Fund, December 18, 2000. http://www.imf.org/external/np/vc/2000/121800.htm.
22. Wall Street Journal Forecasting panel, February 2009. http://online.wsj.com/article/SB123445757254678091.html.
23. Torsten Rieke, “Ganz oben in der Wall Street,” Handelsblatt, October 19, 2005. http://www.handelsblatt.com/unternehmen/management/koepfe/ganz-oben-in-der-wall-street/2565624.html.
24. “Federal Reserve Economic Data,” Economic Research, Federal Reserve Bank of St. Louis. http://research.stlouisfed.org/fred2/.
25. Lakshman Achuthan and Anirvan Benerji, Beating the Business Cycle: How to Predict and Profit from Turning Points in the Economy New York: Random House, 2004). Kindle edition, locations 1476–1477.
26. “U.S. Business Cycle Expansions and Contractions,” National Bureau of Economic Research. http://www.nber.org/cycles.html.
27. Specifically, the stock market as measured by the S&P 500.
28. The original National Football League included the Pittsburgh Steelers, Baltimore Colts (now the Indianapolis Colts), and the Cleveland Browns (the original version of which became the Baltimore Ravens). The way the indicator is usually defined, these teams are counted as belonging to the original National Football Conference even though they have since moved to the American Football Conference, the successor of the American Football League. The fact that this slightly archaic definition is used is another tip-off that the indicator is contrived.
29. For instance, explaining the amount of stock market growth through a simple regression model that uses the conference affiliation of the Super Bowl winner and a constant term as its only inputs would yield this estimate.
30. “Powerball—Prizes and Odds,” Multi-State Lottery Association. http://www.powerball.com/powerball/pb_prizes.asp.
31. Achuthan and Benerji, Beating the Business Cycle, Kindle location 1478.
32. Gene Sperling, “The Insider’s Guide to Economic Forecasting,” Inc. Magazine, August 1, 2003. http://www.inc.com/magazine/20030801/forecasting_pagen_3.html.
33. Ibid.
34. Douglas M. Woodham, “Are the Leading Indicators Signaling a Recession?” Federal Reserve Bank of New York Review (Autumn 1984). http://www.newyorkfed.org/research/quarterly_review/1984v9/v9n3article8.pdf.
35. By “real time,” I mean based on the values of the Leading Economic Index as available to forecasters at the time, before revisions to the data and the composition of the index. See Francis X. Diebold and Glenn D. Rudebusch, “Forecasting Output with the Composite Leading Index: A Real-Time Analysis,” Journal of the American Statistical Association 86, 415 (September 1991), pp. 603–610.
36. Mark J. Perry, “Consumer Confidence Is a Lagging Indicator: Expect Post-Recession Gloom Through 2010,” Seeking Alpha, October 29, 2009. http://seekingalpha.com/article/169740-consumer-confidence-is-a-lagging-indicator-expect-post-recession-gloom-through-2010.
37. Robert Lucas, “Econometric Policy Evaluation: A Critique,” and Karl Brunner and A. Meltzer, “The Phillips Curve and Labor Markets,” Carnegie-Rochester Conference Series on Public Policy, American Elsevier, 1976, pp. 19–46. http://pareto.uab.es/mcreel/reading_course_2006_2007/lucas1976.pdf.
38. C.A.E. Goodhart, “Problems of Monetary Management: The U.K. Experience,” Papers in Monetary Economics, Reserve Bank of Australia, 1975.
39. The term that economists use for this condition is exogeneity.
40. Job growth as measured by the percentage change in net nonfarm payrolls.
41. “Year” in this instance refers to the change from the second quarter of 2009 through the first quarter of 2010.
42. The National Bureau of Economic Research, “U.S. Business Cycle Expansions and Contractions.”
43. “Japan: Gross Domestic Product, constant prices (National currency),” Global Insight and Nomura database via International Monetary Fund, last updated 2010. http://www.imf.org/external/pubs/ft/weo/2011/02/weodata/weorept.aspx?pr.x=38&pr.y=9&sy=1980&ey=2016&scsm=1&ssd=1&sort=country&ds=.&br=1&c=158&s=NGDP_R&grp=0&a=.
44. “Minutes of the Federal Open Market Committee;” Federal Reserve System, October 30–31, 2007. http://www.federalreserve.gov/monetarypolicy/files/fomcminutes20071031.pdf.
45. “Gauging the Uncertainty of the Economic Outlook from Historical Forecasting Errors,” by David Reifschneider and Peter Tulip; Finance and Economics Discussion Series, Divisions of Research and Statistics and Monetary Affairs, Federal Reserve Board, November 19, 2007. http://www.federalreserve.gov/Pubs/FEDS/2007/200760/200760pap.pdf.
46. The government can do a better job of estimating incomes once it sees tax filings (Americans, as much as they might dislike taxes, are relatively honest about paying them). But income earned in January 2009 would not be reported to the IRS until April 15, 2010. Then the IRS might take a few more months to collect all the data and report it to the Bureau of Economic Analysis. So although this is highly useful information, it might become available only after a lag of eighteen months to two years—much too late to be of any use to forecasters. Nevertheless, the government continues to refine its estimates of indicators like GDP for years after the fact through what are known as benchmark revisions.
47. “Historical Data Files for the Real-Time Data Set: Real GNP/GDP (ROUTPUT),” Federal Reserve Bank of Philadelphia. http://www.philadelphiafed.org/research-and-data/real-time-center/real-time-data/data-files/ROUTPUT/.
48. Specifically, the 95 percent margin of error.
49. The opposite case, happily, can sometimes arise as well. The government initially reported negative growth in the third quarter of 1981. But the data now says that the economy grew at nearly a 5 percent clip.
50. Although economists often do not give enough attention to the distinction between real time and revised data when they present their forecasts. Revisions tend to bring different economic indicators more in line with one another. But the data is much messier when it is revealed in real time; in the spring of 2012, for instance, some economic indicators (like personal income) were near-recessionary while others (like industrial production) were suggestive of strong growth. In a few years, the data from this period will probably look much cleaner and will tell a more consistent story; the personal income figures will be revised upward some or the industrial production numbers downward. But that is much too late for economists trying to make a forecast. Building a forecast model from revised data will lead you to overestimate the ease of the task of forecasting.
51. This is not just me being glib. One can plot the error made in annual GDP predictions by the Survey of Professional Forecasters against a time trend and find that there has been no overall improvement since 1968.
52. “U.S. Economy Tipping into Recession,” Economic Cycle Research Institute, September 30, 2011. http://www.businesscycle.com/reports_indexes/reportsummarydetails/1091.
53. Chris Isidore, “Forecast Says Double-Dip Recession Is Imminent,” CNNMoney; September 30, 2011. http://money.cnn.com/2011/09/30/news/economy/double_dip_recession/index.htm.
54. Economic Cycle Research Institute, “U.S. Economy Tipping into Recession,” September 30, 2011. http://www.businesscycle.com/reports_indexes/reportsummarydetails/1091.
55. Achuthan and Benerji, Beating the Business Cycle, Kindle locations 192–194.
56. Chris Anderson, “The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,” Wired magazine, 16.07; June 23, 2008. http://www.wired.com/science/discoveries/magazine/16-07/pb_theory.
57. I don’t publish economic forecasts, but I certainly would not claim that I was especially bullish at the time.
58. A variety of conceptually similar methods based on “leading indicators” were predicting considerable growth at the time, or at least very little chance of a rec
ession. See Dwaine van Vuuren, “U.S. Recession—an Opposing View,” Advisor Perspectives, January 3, 2012. http://www.advisorperspectives.com/newsletters12/US_Recession-An_Opposing_View.php. Although ECRI’s methodology is a little opaque, it appears that their indices may have placed a lot of emphasis on commodities prices, which were declining in late 2011 after having been in something of a bubble.
59. From September 30, 2011 (S&P 500 closed at 1131.42) to March 30, 2012 (S&P closed at 1379.49).
60. Henry Blodget, “ECRI’s Lakshman Achuthan: No, I’m Not Wrong—We’re Still Headed for Recession,” Daily Ticker, Yahoo! Finance; May 9, 2012. http://finance.yahoo.com/blogs/daily-ticker/ecri-lakshman-achuthan-no-m-not-wrong-still-145239368.html.
61. In the November forecasts made from 1968 through 2009 of the next year’s GDP in the Survey of Professional Forecasters, the root mean square error (RMSE) for an individual economist’s forecast was 2.27 points, while the RMSE of the aggregate forecast was 1.92 points. Thus, averaging the forecasts reduced error by about 18 percent.
62. Stephen K. McNees, “The Role of Judgment in Macroeconomic Forecasting Accuracy,” International Journal of Forecasting, 6, no. 3, pp. 287–99, October 1990. http://www.sciencedirect.com/science/article/pii/016920709090056H.
63. About the only economist I am aware of who relies solely on statistical models without applying any adjustments to them is Ray C. Fair of Yale. I looked at the accuracy of the forecasts from Fair’s model, which have been published regularly since 1984. They aren’t bad in some cases: the GDP and inflation forecasts from Fair’s model have been roughly as good as those of the typical judgmental forecaster. However, the model’s unemployment forecasts have always been very poor, and its performance has been deteriorating recently as it considerably underestimated the magnitude of the recent recession while overstating the prospects for recovery. One problem with statistical models is that they tend to perform well until one of their assumptions is violated and they encounter a new situation, in which case they may produce very inaccurate forecasts. In this case, a global financial crisis would represent a new situation to a model that had been “trained” on economic data from after World War II since none had occurred during that period.