The Signal and the Noise

by Nate Silver

  Results like these ought to be challenging to anyone who takes a caricatured view of climate science. They should cut against the notion that scientists are injudiciously applying models to make fantastical predictions about the climate; instead, the scientists have as much doubt about the models as many of their critics.43 However, cinematographic representations of climate change, like Al Gore’s An Inconvenient Truth, have sometimes been less cautious, portraying a polar bear clinging to life in the Arctic, or South Florida and Lower Manhattan flooding over.44 Films like these are not necessarily a good representation of the scientific consensus. The issues that climate scientists actively debate are much more banal: for instance, how do we develop computer code to make a good representation of a cloud?

  Climate Science and Complexity

  Weather forecasters and climatologists often find themselves at odds;45 a large number of meteorologists are either implicitly or explicitly critical of climate science.

  Weather forecasters have endured decades of struggle to improve their forecasts, and they can still expect to receive angry e-mails whenever they get one wrong. It is challenging enough to predict the weather twenty-four hours in advance. So how can climate forecasters, who are applying somewhat analogous techniques, expect to predict what the climate will look like decades from now?

  Some of the distinction, as in the case of the term consensus, is semantic. Climate refers to the long-term equilibriums that the planet achieves; weather describes short-term deviations from it.46 Climate forecasters are not attempting to predict whether it will rain in Tulsa on November 22, 2062, although they are perhaps interested in whether it will be rainier on average throughout the Northern Hemisphere.

  Meteorologists, nevertheless, have to wrestle with complexity:* the entire discipline of chaos theory developed out of what were essentially frustrated attempts to make weather forecasts. Climatologists have to deal with complexity as well: clouds, for instance, are small-scale phenomena that require a lot of computer power to model accurately, but they can have potentially profound effects on the feedback loops intrinsic to climate forecasts.47

  The irony is that weather forecasting is one of the success stories in this book. Through hard work, and a melding of computer power with human judgment, weather forecasts have become much better than they were even a decade or two ago. Given that forecasters in most domains are prone to overconfidence, it is admirable that weather forecasters are hard on themselves and their forecasting peers. But the improvements they have made refute the idea that progress is hopeless in the face of complexity.

  The improvements in weather forecasts are a result of two features of their discipline. First meteorologists get a lot of feedback—weather predictions play out daily, a reality check that helps keep them well-calibrated. This advantage is not available to climate forecasters and is one of the best reasons to be skeptical about their predictions, since they are made at scales that stretch out to as many as eighty or one hundred years in advance.

  Meteorologists also benefit, however, from a strong understanding of the physics of the weather system, which is governed by relatively simple and easily observable laws. Climate forecasters potentially have the same advantage. We can observe clouds and we have a pretty good idea of how they behave; the challenge is more in translating that into mathematical terms.

  One favorable example for climate forecasting comes from the success at forecasting the trajectories of some particularly big and important clouds—those that form hurricanes. Emanuel’s office at MIT, designated as room 54-1814, is something of a challenge to find (I was assisted by an exceptional janitor who may as well have been the inspiration for Good Will Hunting). But it offers a clear view of the Charles River. It was easy to imagine a hurricane out in the distance: Would it careen toward Cambridge or blow out into the North Atlantic?

  Emanuel articulated a distinction between two types of hurricane forecasts. One is purely statistical. “You have a long record of the phenomenon you’re interested in. And you have a long record of what you consider to be viable predictors—like the wind in a large-scale flow of the atmosphere, or the temperature of the ocean, what have you,” he said. “And without being particularly physical about it, you just use statistics to relate to what you’re trying to predict to those predictors.”

  Imagine that a hurricane is sitting in the Gulf of Mexico. You could build a database of past hurricanes and look at their wind speed and their latitude and longitude and the ocean temperature and so forth, and identify those hurricanes that were most similar to this new storm. How had those other hurricanes behaved? What fraction struck populous areas like New Orleans and what fraction dissipated? You would not really need all that much meteorological knowledge to make such a forecast, just a good database.

  Techniques like these can provide for crude but usable forecasts. In fact, up until about thirty years ago, purely statistical models were the primary way that the weather service forecasted hurricane trajectories.

  Such techniques, however, are subject to diminishing returns. Hurricanes are not exactly rare, but severe storms hit the United States perhaps once every year on average. Whenever you have a large number of candidate variables applied to a rarely occurring phenomenon, there is the risk of overfitting your model and mistaking the noise in the past data for a signal.

  There is an alternative, however, when you have some knowledge of the structure behind the system. This second type of model essentially creates a simulation of the physical mechanics of some portion of the universe. It takes much more work to build than a purely statistical method and requires a more solid understanding of the root causes of the phenomenon. But it is potentially more accurate. Models like these are now used to forecast hurricane tracks and they have been highly successful. As I reported in chapter 4, there has been roughly a threefold increase in the accuracy of hurricane track projections since the 1980s, and the location near New Orleans where Hurricane Katrina made landfall had been pinpointed well more than forty-eight hours in advance48 (though not everyone chose to listen to the forecast). Statistically driven systems are now used as little more than the baseline to measure these more accurate forecasts against.

  Beyond a Cookbook Approach to Forecasting

  The criticisms that Armstrong and Green make about climate forecasts derive from their empirical study of disciplines like economics in which there are few such physical models available49 and the causal relationships are poorly understood. Overly ambitious approaches toward forecasting have often failed in these fields, and so Armstrong and Green infer that they will fail in climate forecasting as well.

  The goal of any predictive model is to capture as much signal as possible and as little noise as possible. Striking the right balance is not always so easy, and our ability to do so will be dictated by the strength of the theory and the quality and quantity of the data. In economic forecasting, the data is very poor and the theory is weak, hence Armstrong’s argument that “the more complex you make the model the worse the forecast gets.”

  In climate forecasting, the situation is more equivocal: the theory about the greenhouse effect is strong, which supports more complicated models. However, temperature data is very noisy, which argues against them. Which consideration wins out? We can address this question empirically, by evaluating the success and failure of different predictive approaches in climate science. What matters most, as always, is how well the predictions do in the real world.

  I would urge caution against reducing the forecasting process to a series of bumper-sticker slogans. Heuristics like Occam’s razor (“other things being equal, a simpler explanation is better than a more complex one”50) sound sexy, but they are hard to apply. We have seen cases, as in the SIR models used to forecast disease outbreaks, where the assumptions of a model are simple and elegant—but where they are much too naïve to provide for very skillful forecasts. We have also seen cases, as in earthquake prediction, where unbelievably convoluted foreca
sting schemes that look great in the software package fail miserably in practice.

  An admonition like “The more complex you make the model the worse the forecast gets” is equivalent to saying “Never add too much salt to the recipe.” How much complexity—how much salt—did you begin with? If you want to get good at forecasting, you’ll need to immerse yourself in the craft and trust your own taste buds.

  Uncertainty in Climate Forecasts

  Knowing the limitations of forecasting is half the battle, and on that score the climate forecasters do reasonably well. Climate scientists are keenly aware of uncertainty: variations on the term uncertain or uncertainty were used 159 times in just one of the three IPCC 1990 reports.51 And there is a whole nomenclature that the IPCC authors have developed to convey how much agreement or certainty there is about a finding. For instance, the phrase “likely” taken alone is meant to imply at least a 66 percent chance of a prediction occurring when it appears in an IPCC report, while the phrase “virtually certain” implies 99 percent confidence or more.52

  Still, it is one thing to be alert to uncertainty and another to actually estimate it properly. When it comes to something like political polling, we can rely on a robust database of historical evidence: if a candidate is ten points ahead in the polls with a month to go until an election, how often will she wind up winning? We can look through dozens of past elections to get an empirical answer to that.

  The models that climate forecasters build cannot rely on that sort of technique. There is only one planet and forecasts about how its climate will evolve are made at intervals that leap decades into the future. Although climatologists might think carefully about uncertainty, there is uncertainty about how much uncertainty there is. Problems like these are challenging for forecasters in any discipline.

  Nevertheless, it is possible to analyze the uncertainty in climate forecasts as having three component parts. I met with Gavin Schmidt, a NASA colleague of Hansen’s and a somewhat sarcastic Londoner who is a co-author of the blog RealClimate.org, at a pub near his office in Morningside Heights in New York.

  Schmidt took out a cocktail napkin and drew a graph that looked something like what you see in figure 12-3, which illustrates the three distinct problems that climate scientists face. These different types of uncertainty become more or less prevalent over the course of a climate forecast.

  FIGURE 12-3: SCHEMATIC OF UNCERTAINTY IN GLOBAL WARMING FORECASTS

  First, there is what Schmidt calls initial condition uncertainty—the short-term factors that compete with the greenhouse signal and impact the way we experience the climate. The greenhouse effect is a long-term phenomenon, and it may be obscured by all types of events on a day-to-day or year-to-year basis.

  The most obvious type of initial condition uncertainty is simply the weather; when it comes to forecasting the climate, it represents noise rather than signal. The current IPCC forecasts predict that temperatures might increase by 2°C (or about 4°F) over the course of the next century. That translates into an increase of just 0.2°C per decade, or 0.02°C per year. Such a signal is hard to perceive when temperatures can easily fluctuate by 15°C from day to night and perhaps 30°C from season to season in temperate latitudes.

  In fact, just a few days before I met with Schmidt in 2011, there had been a freakish October snowstorm in New York and other parts of the Northeast. The snowfall, 1.3 inches in Central Park, set an October record there,53 and was more severe in Connecticut, New Jersey, and Massachusetts, leaving millions of residents without power.54

  Central Park happens to have a particularly good temperature record;55 it dates back to 1869.56 In figure 12-4, I have plotted the monthly average temperature for Central Park in the century encompassing 1912 through 2011. You will observe the seasons in the graphic; the temperature fluctuates substantially (but predictably enough) from warm to cool and back again—a little more so in some years than others. In comparison to the weather, the climate signal is barely noticeable. But it does exist: temperatures have increased by perhaps 4°F on average over the course of this one-hundred-year period in Central Park.

  FIGURE 12-4: CENTRAL PARK (NEW YORK CITY) MONTHLY AVERAGE TEMPERATURES, 1912–2011, IN °F

  There are also periodic fluctuations that take hold at periods of a year to a decade at a time. One is dictated by what is called the ENSO cycle (the El Niño–Southern Oscillation). This cycle, which evolves over intervals of about three years at a time,57 is instigated by temperature shifts in the waters of the tropical Pacific. El Niño years, when the cycle is in full force, produce warmer weather in much of the Northern Hemisphere, and probably reduce hurricane activity in the Gulf of Mexico.58 La Niña years, when the Pacific is cool, do just the opposite. Beyond that, relatively little is understood about the ENSO cycle.

  Another such medium-term process is the solar cycle. The sun gives off slightly more and slightly less radiation over cycles that last for about eleven years on average. (This is often measured through sunspots, the presence of which correlate with higher levels of solar activity.) But these cycles are somewhat irregular: Solar Cycle 24, for instance, which was expected to produce a maximum of solar activity (and therefore warmer temperatures) in 2012 or 2013, turned out to be somewhat delayed.59 Occasionally, in fact, the sun can remain dormant for decades at a time; the Maunder Minimum, a period of about seventy years during the late seventeenth and early eighteenth centuries when there was very little sunspot activity, may have triggered cooler temperatures in Europe and North America.60

  Finally, there are periodic interruptions from volcanoes, which blast sulfur—a gas that has an anti-greenhouse effect and tends to cool the planet— into the atmosphere. The eruption of Mount Pinatubo in 1991 reduced global temperatures by about 0.2°C for a period of two years, equivalent to a decade’s worth of greenhouse warming.

  The longer your time horizon, the less concern you might have about these medium-term effects. They can dominate the greenhouse signal over periods of a year to a decade at a time, but they tend to even out at periods beyond that.

  Another type of uncertainty, however—what Schmidt calls scenario uncertainty—increases with time. This concerns the level of CO2 and other greenhouse gases in the atmosphere. At near time horizons, atmospheric composition is quite predictable. The level of industrial activity is fairly constant, but CO2 circulates quickly into the atmosphere and remains there for a long time. (Its chemical half-life has been estimated at about thirty years.61) Even if major industrialized countries agreed to immediate and substantial reductions in CO2 emissions, it would take years to reduce the growth rate of CO2 in the atmosphere, let alone to actually reverse it. “Neither you nor I will ever see a year in which carbon dioxide concentrations have gone down, not ever,” Schmidt told me. “And not your children either.”

  Still, since climate models rely on specific assumptions about the amount of atmospheric CO2, this can significantly complicate forecasts made for fifty or one hundred years out and affect them at the margin in the nearer term, depending on how political and economic decisions influence CO2 emissions.

  Last, there is the structural uncertainty in the models. This is the type of uncertainty that both climate scientists and their critics are rightly most worried about, because it is the most challenging to quantify. It concerns how well we understand the dynamics of the climate system and how well we can represent them mathematically. Structural uncertainty might increase slightly over time, and errors can be self-reinforcing in a model of a dynamic system like the climate.

  Taken together, Schmidt told me, these three types of uncertainty tend to be at a minimum at a period of about twenty or twenty-five years in advance of a climate forecast. This is close enough that we know with reasonable certainty how much CO2 there will be in the atmosphere—but far enough away that the effects of ENSO and volcanoes and the solar cycle should have evened out.

  As it happens, the first IPCC report, published in 1990, falls right into this twenty-year swe
et spot. So do some of the early forecasts made by James Hansen in the 1980s. It is time, in other words, to assess the accuracy of the forecasts. So how well did they do?

  A Note on the Temperature Record

  To measure the accuracy of a prediction, you first need a measuring stick—and climate scientists have quite a few choices. There are four major organizations that build estimates of global temperatures from thermometer readings at land and sea stations around the globe. These organizations include NASA62 (which maintains its GISS63 temperature record), NOAA64 (the National Oceanic and Atmospheric Administration, which manages the National Weather Service), and the meteorological offices of the United Kingdom65 and Japan.66

  A more recent entrant into the temperature sweepstakes are observations from satellites. The most commonly used satellite records are from the University of Alabama at Huntsville and from a private company called Remote Sensing Systems.67 The satellites these records rely on do not take the temperature directly—instead, they infer it by measuring microwave radiation. But the satellites’ estimates of temperatures in the lower atmosphere68 provide a reasonably good proxy for surface temperatures.69

  FIGURE 12-5: GLOBAL TEMPERATURE ANOMALY RELATIVE TO 1951–80 BASELINE: SIX TEMPERATURE RECORDS

  The temperature records also differ in how far they track the climate backward; the oldest are the observations from the UK’s Met Office, which date back to 1850; the satellite records are the youngest and date from 1979. And the records are measured relative to different baselines—the NASA/GISS record is taken relative to average temperatures from 1951 through 1980, for instance, while NOAA’s temperatures are measured relative to the average throughout the twentieth century. But this is easy to correct for,70 and the goal of each system is to measure how much temperatures are rising or falling rather than what they are in any absolute sense.