The Signal and the Noise

by Nate Silver

  25. “Election Results: House Big Board,” New York Times, November 2, 2010. http://elections.nytimes.com/2010/results/house/big-board.

  26. Nate Silver, “A Warning on the Accuracy of Primary Polls,” FiveThirtyEight, New York Times, March 1, 2012. http://fivethirtyeight.blogs.nytimes.com/2012/03/01/a-warning-on-the-accuracy-of-primary-polls/.

  27. Nate Silver, “Bill Buckner Strikes Again,” FiveThirtyEight, New York Times; September 29, 2011. http://fivethirtyeight.blogs.nytimes.com/2011/09/29/bill-buckner-strikes-again/.

  28. Otherwise, you should have assigned the congressman a 100 percent chance of victory instead.

  29. Matthew Dickinson, “Nate Silver Is Not a Political Scientist,” in Presidential Power: A Nonpartisan Analysis of Presidential Power, Blogs Dot Middlebury, November 1, 2010. http://blogs.middlebury.edu/presidentialpower/2010/11/01/nate-silver-is-not-a-political-scientist/.

  30. Sam Wang, “A Weakness in FiveThirtyEight.com,” Princeton Election Consortium, August 8, 2008. http://election.princeton.edu/2008/08/04/on-a-flaw-in-fivethirtyeightcom/.

  31. Douglas A. Hibbs Jr., “Bread and Peace Voting in U.S. Presidential Elections,” Public Choice 104 (January 10, 2000): pp. 149–180. http://www.douglas-hibbs.com/HibbsArticles/Public%20Choice%202000.pdf.

  32. Hibbs’s model predicted Al Gore to win 54.8 percent of the two-party vote (that is, excluding votes for third-party candidates) when Gore in fact won 50.3 percent—an error of 4.5 percent. His model claimed to have a standard error of about 2 points in predicting any one candidate’s vote share (or about 4 points when predicting the margin between them). His forecast thus overestimated Gore’s standing by 2.25 standard deviations, something that should occur only about 1 in 80 times according to the normal distribution.

  33. James E. Campbell, “The Referendum That Didn’t Happen: The Forecasts of the 2000 Presidential Election,” PS: Political Science & Politics (March 2001). http://cas.buffalo.edu/classes/psc/fczagare/PSC%20504/Campbell.pdf.

  34. Andrew Gelman and Gary King, “Why Are American Presidential Election Campaign Polls So Predictable?” British Journal of Political Science 23, no. 4 (October 1993). http://www.rochester.edu/College/faculty/mperess/ada2007/Gelman_King.pdf.

  35. Nate Silver, “Models Based on ‘Fundamentals’ Have Failed at Predicting Presidential Elections,” FiveThirtyEight, New York Times, March 26, 2012.

  36. Between 1998 and 2008, the average poll for a U.S. Senate race conducted in the final three weeks of the campaign missed by 5.0 points, whereas the average poll for a U.S. House race missed by 5.8 points.

  37. One mild criticism of Cook Political’s methodology is that they classify too many races as Toss-Up, even where there is probably enough evidence to posit a modest advantage for one or another of the candidates. FiveThirtyEight’s methodology, which identifies a nominal favorite in all races no matter how slim his apparent advantage, correctly identified the winner in thirty-eight of the fifty races (76 percent) that Cook Political characterized as Toss-Up in 2010.

  38. Between 1998 and 2010, there were seventeen instances in which Cook classified a race one way (favoring the Democrat, for instance) while the average of polls came to the opposite conclusion (perhaps finding a narrow advantage for the Republican). The Cook forecasts made the right call in thirteen of those seventeen cases.

  39. Kapanke was later ousted from his State Senate seat in the Wisconsin recall elections of 2011.

  40. Scott Schneider, “Democrats Unfairly Accuse Dan Kapanke of Ethics Violations,” La Crosse Conservative Examiner, August 27, 2010. http://www.examiner.com/conservative-in-la-crosse/democrats-unfairly-accuse-dan-kapanke-of-ethics-violations.

  41. Cook Political eventually moved its rating to Lean Democrat from Likely Democrat several weeks later, but on account of the Democrats’ deteriorating situation in the national political environment rather than anything that had happened during Kapanke’s interview.

  42. Paul E. Meehl, “When Shall We Use Our Heads Instead of the Formula,” Journal of Counseling Psychology 4, no. 4 (1957), pp. 268–273. http://mcps.umn.edu/assets/pdf/2.10_Meehl.pdf.

  43. Douglas Harper, Online Etymology Dictionary. http://www.etymonline.com/index.php?term=objective.

  CHAPTER 3: ALL I CARE ABOUT IS W’S AND L’S

  1. Nate Silver in Jonah Keri, et al., Baseball Between the Numbers: Why Everything You Know About the Game Is Wrong (New York: Basic Books, 2006).

  2. Danny Knobler, “The Opposite of a ‘Tools Guy,’ Pedroia’s Simply a Winner,” CBSSports.com, November 18, 2008. http://www.cbssports.com/mlb/story/11116048.

  3. Nate Silver, “Lies, Damned Lies: PECOTA Takes on Prospects, Wrap-up,” BaseballProspectus.com, March 8, 2006. http://www.baseballprospectus.com/article.php?articleid=4841.

  4. Law is also a former writer at Baseball Prospectus.

  5. Keith Law, “May Rookies Struggling to Show They Belong,” ESPN.com, May 12, 2007. http://insider.espn.go.com/mlb/insider/columns/story?columnist=law_keith&id=2859877.

  6. Pedroia’s day-by-day statistics are per Baseball-Reference.com.

  7. Tommy Craggs, “Dustin Pedroia Comes out Swinging,” Boston Magazine, April 2009. http://www.bostonmagazine.com/articles/dustin_pedroia/page5.

  8. This calculation is based on the binomial distribution and assumes that a player gets five hundred at-bats.

  9. Or, more precisely, the typical hitter. Pitchers follow different and more irregular aging patterns.

  10. More particularly, a hitter’s age-32 season seems to be about the point at which his skills begin to decline at an accelerated rate.

  11. Jeff Sonas, “The Greatest Chess Player of All Time, Part II, Chessbase.com, April 28, 2004. http://www.chessbase.com/newsdetail.asp?newsid=2354.

  12. Bruce Weinberg and David Galenson, “Creative Careers: The Life Cycles of Nobel Laureates in Economics,” NBER Working Paper No. 11799, National Bureau of Economic Research, November 2005. http://www.econ.brown.edu/econ/sthesis/IanPapers/tcl.html.

  13. Del Jones, “Does Age Matter When You’re CEO?” USA Today, September 11, 2008. http://www.usatoday.com/money/companies/management/2008-08-12-obama-mccain-age-ceos_N.htm.

  14. Gary Huckabay, “6-4-3,” Baseball Prospectus, August 2, 2002. http://www.baseballprospectus.com/article.php?articleid=1581.

  15. Arlo Lyle, “Baseball Prediction Using Ensemble Learning,” thesis submitted to the University of Georgia, 2007. http://www.ai.uga.edu/IAI/Theses/lyle_arlo.pdf.

  16. Bill James, “Whatever Happened to the Hall of Fame,” Fireside (1995): p. 89.

  17. Note, however, that a significant amount of the back-end processing for PECOTA occurred in a statistical language called STATA.

  18. I chose World War II as a cutoff point because a number of developments occurred shortly after the war that made professional baseball the modern game that it is today: the breaking of the color barrier by Jackie Robinson (1947); the first televised World Series (1947); the movement of teams to the West Coast (1957); the introduction of night baseball, which had initially occurred as early as 1935 but gained more currency during the war as workers put in long hours at factories and needed some recreation at night.

  19. Alan Schwarz, “The Great Debate,” Baseball America, January. 7, 2005. http://www.baseballamerica.com/today/features/050107debate.html.

  20. Per interview with Billy Beane.

  21. Nate Silver, “What Tim Geithner Can Learn from Baseball,” Esquire, March 11, 2009. http://www.esquire.com/features/data/mlb-player-salaries-0409.

  22. As a result of my original agreement in 2003 and a subsequent agreement in 2009, Baseball Prospectus now fully owns and operates PECOTA. Beginning with the 2010 season, the PECOTA forecasts reflect certain changes, improvements, and departures from my original methodology. The methods I describe herein apply to the 2003–2009 version of PECOTA specifically.

  23. Nate Silver, “PECOTA Takes on the Field,” Baseball Prospectus, January 16, 2004. http://www.baseballprospectus.com/article.php
?articleid=2515.

  24. Nate Silver, “Lies, Damned Lies: Projection Reflection,” Baseball Prospectus, October 11, 2006. http://www.baseballprospectus.com/article.php?articleid=5609.

  25. Ibid.

  26. Dave Van Dyck, “Computer Crashes White Sox,” Chicago Tribune, March 11, 2007. http://articles.chicagotribune.com/2007-03-11/sports/0703110075_1_computer-paul-konerko-projections.

  27. Steve Slowinski, “The Projection Rundown: The Basics on Marcels, ZIPS, CAIRO, Oliver, and the Rest,” FanGraphs.com, February 16, 2011. http://www.fangraphs.com/library/index.php/the-projection-rundown-the-basics-on-marcels-zips-cairo-oliver-and-the-rest/.

  28. Silver, “Lies, Damned Lies: PECOTA Takes on Prospects, Wrap-up.”

  29. There are many different versions of WARP and similar statistics. Naturally, I use the Baseball Prospectus version for these calculations.

  30. Dave Cameron, “Win Values Explained: Part Six,” FanGraphs.com, January 2, 2009. http://www.fangraphs.com/blogs/index.php/win-values-explained-part-six/.

  31. Silver, “Lies, Damned Lies: PECOTA Takes on Prospects, Introduction,” Baseball Prospectus, February 1, 2007. http://www.baseballprospectus.com/article.php?articleid=5836.

  32. “All-Time Top 100 Prospects,” Baseball America. http://www.baseballamerica.com/today/prospects/rankings/top-100-prospects/all-time.html.

  33. “1997 Oakland Athletics Batting, Pitching, & Fielding Statistics,” Baseball-Reference.com. http://www.baseball-reference.com/teams/OAK/1997.shtml

  34. It should be noted, however, that with the average major-league payroll now at about $100 million, a forecasting system that allowed a baseball team to spend its money 2 percent more efficiently would save them $2 million. Compared with the five-figure salaries that statistical analysts receive in major league front offices, that’s quite a bargain.

  35. “Detroit Tigers 11, Kansas City Athletics 4: Game Played on Tuesday, April 13, 1965 (D) at Municipal Stadium,” Retrosheet.org. http://www.retrosheet.org/boxesetc/1965/B04130KC.11965.htm.

  36. “John Sanders, Grand Island,” Inductee 2002, Nebraska High School Sports Hall of Fame Foundation. http://www.nebhalloffame.org/2002/sanders.htm.

  37. Steve Treder, “Cash in the Cradle: The Bonus Babies,” The Hardball Times, November 1, 2004. http://www.hardballtimes.com/main/article/cash-in-the-cradle-the-bonus-babies/.

  38. Mike Pesca, “The Man Who Made Baseball’s Box Score a Hit,” National Public Radio, July 30, 2009. http://www.npr.org/templates/story/story.php?storyId=106891539.

  39. Why teams persisted in believing the contrary for so long is a good question. It may have had to do with the fact that a walk was traditionally perceived as a mistake made by the pitcher, rather than a skill exhibited by the batter. It may have been that a walk is perceived as too passive in a culture that prizes machismo. But the industry has grown wise to the value of OBP, which is now the category most highly correlated with salaries paid to free agents. So the A’s can no longer exploit the inefficiency—after years of finishing near the top of the league in walks drawn, they placed just tenth out of fourteen American League teams in the category in 2009.

  40. Ken C. Winters, “Adolescent Brain Development and Drug Abuse,” Treatment Research Institute, November 2004. http://www.factsontap.org/docs/2004Nov_AdolescentBrain.pdf.

  41. Per interview with John Sanders.

  42. Players spend their first two major-league seasons subject to something called the reserve clause, which dictates that they may not sign with any other clubs. This means that the player has almost no leverage at all, and is usually signed for something close to the league minimum salary of $400,000. After that, he spends his third through sixth years subject to arbitration, in which both he and the team submit salary requests, and a three-person panel decides which is the more appropriate. But salaries awarded in arbitration are typically only about 60 percent of those secured by free agents with comparable skills, so the teams are getting these players at a substantial discount.

  43. Nor is there any reason that you couldn’t have scouts rate a player’s mental tools along with his physical ones.

  44. Jeremy Greenhouse, “Touching Bases: Best PITCHf/x Pitches of 2009,” Baseball Analysts; March 4, 2010. http://baseballanalysts.com/archives/2010/03/top_pitchfx_pit.php.

  45. Ibid.

  46. “Baseball Hall of Fame Second Basemen,” Baseball Almanac. http://www.baseball-almanac.com/hof/hofst2b.shtml.

  47. Some of the conversation with James is also taken from a subsequent phone call I made to him.

  CHAPTER 4: FOR YEARS YOU’VE BEEN TELLING US THAT RAIN IS GREEN

  1. Forecaster Stewart, “Tropical Depression Twelve: ZCZC MIATCDAT2 ALL, TTAA00 KNHC DDHHMM,” National Hurricane Center, National Weather Service, August. 23, 2005. http://www.nhc.noaa.gov/archive/2005/dis/al122005.discus.001.shtml?

  2. Based on statistics from StormPulse.com from the 2000 through 2011 Atlantic Basin hurricane seasons. The exact figure for the percentage of tropical depressions that became hurricanes during this period was 43 percent, while 88 percent of tropical depressions became at least tropical storms.

  3. Stewart, “Tropical Storm Katrina: ZCZC MIATCDAT2 ALL, TTAA00 KNHC DDHHMM,” August 24, 2005. http://www.nhc.noaa.gov/archive/2005/dis/al122005.discus.005.shtml?

  4. By convention, cyclones do not receive names until they become tropical storms with sustained wind speeds of at least 39 mph. They become hurricanes when their wind increases to 74 mph. So Tropical Depression Twelve was briefly Tropical Storm Katrina before becoming Hurricane Katrina.

  5. Forecaster Knabb, “Hurricane Katrina: : ZCZC MIATCDAT2 ALL, TTAA00 KNHC DDHHMM,” National Hurricane Center, National Weather Service; August 27, 2005. http://www.nhc.noaa.gov/archive/2005/dis/al122005.discus.016.shtml?

  6. “Washing Away—Special Report from the Times-Picayune,” Times-Picayune, June 23–27, 2002. http://www.nola.com/hurricane/content.ssf?/washingaway/index.html.

  7. Ezra Boyd, “The Evacuation of New Orleans for Hurricane Katrina: A Synthesis of the Available Data,” presentation for the National Evacuation Conference, February 5, 2010. http://www.nationalevacuationconference.org/files/presentations/day2/Boyd_Ezra.pdf.

  8. “Survey of Hurricane Katrina Evacuees,” Washington Post, Harvard University, and the Kaiser Family Foundation, September 2005. http://www.kff.org/newsmedia/upload/7401.pdf.

  9. “The Weatherman,” Curb Your Enthusiasm, season 4, episode 4, HBO, January. 25, 2004.

  10. joesixpacker, “The Mitt Romney Weathervane,” YouTube, December. 24, 2011. http://www.youtube.com/watch?v=PWPxzDd661M.

  11. Willis I. Milham, Meteorology. A Text-Book on the Weather, the Causes of Its Changes, and Weather Forecasting for the Student and General Reader (New York: Macmillan, 1918).

  12. Aristotle, Meteorology, translated by E. W. Webster. Internet Classics Archive. http://classics.mit.edu/Aristotle/meteorology.html.

  13. Pierre-Simon Laplace, “A Philosophical Essay on Probabilities” (Cosmo Publications, 2007).

  14. The uncertainty principle should not be confused with the observer effect, which is the idea that the act of measuring a system (such as shooting a laser beam at a particle of light) necessarily disrupts it. The two beliefs are not inherently incompatible—but the uncertainty principle is a stronger statement and is not so satisfyingly intuitive. Indeed, Heisenberg believed his uncertainty principle to be quite counterintuitive. But basically the idea is that, beyond a certain degree of resolution, at the very moment we think we’re able to pin down exactly where a particle is, the particle no longer behaves as a point of matter but instead as something much different: as a wave that is moving. About the most satisfying demonstration I have seen on this is from the MIT physicist Walter Lewin: Acorvettes, “Quantum Mechanics, the Uncertainty Principle, Light Particles,” YouTube, August 4, 2007. http://www.youtube.com/watch?v=KT7xJ0tjB4A.

  15. “London Weather,” in Official London Guide, Visitlondon.com. http://www.visitlondon.com/travel/
weather.

  16. Some of Richardson’s failure was the result of a fairly minor specification error, it was later revealed. If he had corrected for it, he would have produced a reasonably accurate forecast.

  17. J. G. Charney, R. Fjörtoft, and J. von Neumann, “Numerical Integration of the Barotropic Vorticity Equation,” Tellus 2 (1950): pp. 237–254. http://mathsci.ucd.ie/~plynch/eniac/CFvN-1950.pdf.

  18. “Moore’s Law,” Intel Corporation, 2005. ftp://download.intel.com/museum/Moores_Law/Printed_Materials/Moores_Law_2pg.pdf.

  19. Lorenz’s paper was not originally published but instead delivered at a talk to the American Association for the Advancement of Science on December 29, 1972. However, it was later published in Lorenz’s book The Essence of Chaos (Seattle: University of Washington Press, 1995). http://www.washington.edu/uwpress/search/books/LORESS.html.

  20. Douglas Allchin, “Penicillin and Chance,” SHiPS Resource Center. http://www1.umn.edu/ships/updates/fleming.htm.

  21. Per interview with Richard Loft.

  22. 5^5^5 is 298,023,223,876,953,000—about 298 quadrillion. But 5^6^5 is 931,322,574,615,479,000,000, or about 931 quintillion. That “small” mistake would have led us to overestimate the value we were trying to derive by a factor of 3,125.

  23. Yes, calculus is actually useful for something.

  24. NCAR is separate from this part of the bureaucracy, instead run by means of a nonprofit consortium of research universities with funding from the National Science Foundation; this is why it has nicer buildings.

  25. “History of the National Weather Service,” Public Affairs Office, National Weather Service, National Oceanic and Atmospheric Administration, United States Department of Commerce. http://www.weather.gov/pa/history/index.php.

  26. “The Blizzard of 1988,” Nebraska State Historical Society, last updated June 4, 2004. http://www.nebraskahistory.org/publish/markers/texts/blizzard_of_1888.htm.