by Nate Silver
“There’s very little that’s really predictive,” Hatzius told me. “Figuring out what’s truly causal and what’s correlation is very difficult to do.”
Most of you will have heard the maxim “correlation does not imply causation.” Just because two variables have a statistical relationship with each other does not mean that one is responsible for the other. For instance, ice cream sales and forest fires are correlated because both occur more often in the summer heat. But there is no causation; you don’t light a patch of the Montana brush on fire when you buy a pint of Häagen-Dazs.
If this concept is easily expressed, however, it can be hard to apply in practice, particularly when it comes to understanding the causal relationships in the economy. Hatzius noted, for instance, that the unemployment rate is usually taken to be a lagging indicator. And sometimes it is. After a recession, businesses may not hire new employees until they are confident about the prospects for recovery, and it can take a long time to get all the unemployed back to work again. But the unemployment rate can also be a leading indicator for consumer demand, since unemployed people don’t have much ability to purchase new goods and services. During recessions, the economy can fall into a vicious cycle: businesses won’t hire until they see more consumer demand, but consumer demand is low because businesses aren’t hiring and consumers can’t afford their products.
Consumer confidence is another notoriously tricky variable. Sometimes consumers are among the first to pick up warning signs in the economy. But they can also be among the last to detect recoveries, with the public often perceiving the economy to be in recession long after a recession is technically over. Thus, economists debate whether consumer confidence is a leading or lagging indicator,36 and the answer may be contingent on the point in the business cycle the economy finds itself at. Moreover, since consumer confidence affects consumer behavior, there may be all kinds of feedback loops between expectations about the economy and the reality of it.
An Economic Uncertainty Principle
Perhaps an even more problematic set of feedback loops are those between economic forecasts and economic policy. If, for instance, the economy is forecasted to go into recession, the government and the Federal Reserve will presumably take steps to ameliorate the risk or at least soften the blow. Part of the problem, then, is that forecasters like Hatzius have to predict political decisions as well as economic ones, which can be a challenge in a country where the Congress has a 10 percent approval rating.
But this issue also runs a little deeper. As pointed out by the Nobel Prize–winning economist Robert Lucas37 in 1976, the past data that an economic model is premised on resulted in part from policy decisions in place at the time. Thus, it may not be enough to know what current policy makers will do; you also need to know what fiscal and monetary policy looked like during the Nixon administration. A related doctrine known as Goodhart’s law, after the London School of Economics professor who proposed it,38 holds that once policy makers begin to target a particular variable, it may begin to lose its value as an economic indicator. For instance, if the government artificially takes steps to inflate housing prices, they might well increase, but they will no longer be good measures of overall economic health.
At its logical extreme, this is a bit like the observer effect (often mistaken for a related concept, the Heisenberg uncertainty principle): once we begin to measure something, its behavior starts to change. Most statistical models are built on the notion that there are independent variables and dependent variables, inputs and outputs, and they can be kept pretty much separate from one another.39 When it comes to the economy, they are all lumped together in one hot mess.
An Ever-Changing Economy
Even if they could resolve all these problems, economists would still have to contend with a moving target. The American and global economies are always evolving, and the relationships between different economic variables can change over the course of time.
Historically, for instance, there has been a reasonably strong correlation between GDP growth and job growth. Economists refer to this as Okun’s law. During the Long Boom of 1947 through 1999, the rate of job growth40 had normally been about half the rate of GDP growth, so if GDP increased by 4 percent during a year, the number of jobs would increase by about 2 percent.
The relationship still exists—more growth is certainly better for job seekers. But its dynamics seem to have changed. After each of the last couple of recessions, considerably fewer jobs were created than would have been expected during the Long Boom years. In the year after the stimulus package was passed in 2009, for instance, GDP was growing fast enough to create about two million jobs according to Okun’s law.41 Instead, an additional 3.5 million jobs were lost during the period.
Economists often debate about what the change means. The most pessimistic interpretation, advanced by economists including Jeffrey Sachs of Columbia University, is that the pattern reflects profound structural problems in the American economy: among them, increasing competition from other countries, an imbalance between the service and manufacturing sectors, an aging population, a declining middle class, and a rising national debt. Under this theory, we have entered a new and unhealthy normal, and the problems may get worse unless fundamental changes are made. “We were underestimating the role of global change in causing U.S. change,” Sachs told me. “The loss of jobs internationally to China and emerging markets have really jolted the American economy.”
The bigger question is whether the volatility of the 2000s is more representative of the long-run condition of the economy—perhaps the long boom years had been the outlier. During the Long Boom, the economy was in recession only 15 percent of the time. But the rate was more than twice that—36 percent—from 1900 through 1945.42
Although most economists believe that some progress has been made in stabilizing the business cycle, we may have been lucky to avoid more problems. This particularly holds in the period between 1983 and 2006—a subset of the Long Boom that is sometimes called the Great Moderation—when the economy was in recession just 3 percent of the time. But much of the growth was fueled by large increases in government and consumer debt, as well as by various asset-price bubbles. Advanced economies have no divine right to grow at Great Moderation rates: Japan’s, which grew at 5 percent annually during the 1980s, has grown by barely one percent per year since then.43
This may be one reason why forecasters and policy makers were taken so much by surprise by the depth of the 2007 recession. Not only were they failing to account for events like the Great Depression*—they were sometimes calibrating their forecasts according to the Great Moderation years, which were an outlier, historically speaking.
Don’t Throw Out Data
The Federal Open Market Committee, which is charged with setting interest rates, is required by law to release macroeconomic forecasts to Congress at least twice per year. The Fed was in some ways ahead of the curve by late 2007: their forecasts of GDP growth were slightly more bearish than those issued by private-sector forecasters, prompting them to lower interest rates four times toward the end of the year.
Still, in the Fed’s extensive minutes from a late October 2007 meeting, the term “recession” was not used even once in its discussion of the economy.44 The Fed is careful with its language, and the possibility of a recession may nevertheless have been implied through the use of phrases like downside risks. But they were not betting on a recession (their forecast still projected growth), and there was little indication that they were entertaining the possibility of as severe a recession as actually unfolded.
Part of the reason may have been that the Fed was looking at data from the Great Moderation years to set their expectations for the accuracy of their forecasts. In particular, they relied heavily upon a paper that looked at how economic forecasts had performed from 1986 through 2006.45 The problem with looking at only these years is that they contained very little economic volatility: just two relatively mild recessions i
n 1990–1991 and in 2001. “By gauging current uncertainty with data from the mid-1980s on,” the authors warned, “we are implicitly assuming that the calm conditions since the Great Moderation will persist into the future.” This was an awfully big assumption to make. The Fed may have concluded that a severe recession was unlikely in 2007 in part because they had chosen to ignore years in which there were severe recessions.
A forecaster should almost never ignore data, especially when she is studying rare events like recessions or presidential elections, about which there isn’t very much data to begin with. Ignoring data is often a tip-off that the forecaster is overconfident, or is overfitting her model—that she is interested in showing off rather than trying to be accurate.
In this particular case, it was not obvious that economists had improved much at forecasting the business cycle. In figure 6-5a, I’ve compared predicted levels of GDP growth from the Survey of Professional Forecasters against the actual figures for the years 1968 through 1985—these are the years the Fed could have looked at but chose to throw out. You’ll see there’s quite a lot of economic volatility in this period, such as during the inflation-driven recessions of the mid-1970s and early 1980s. Still, the results are not completely discouraging for forecasters, in that the forecasted and actual outcomes have a reasonably strong correlation with one another.
If you make the same plot for the years 1986 through 2006 (as in figure 6-5b), you’ll find just the reverse. Most of the data points—both the forecasted values for GDP and the actual ones—are bunched closely together in a narrow range between about 2 percent and 5 percent annual growth. Because there was so little volatility during this time, the average error in the forecast was less than in the previous period.* However, to the extent there was any variability in the economy, like the mild recessions of 1990–91 or in 2001, the forecasts weren’t doing a very good job of capturing it—in fact, there was almost no correlation between the predicted and actual results. There was little indication that economists had become more skilled at forecasting the course of the economy. Instead, their jobs had become temporarily easier because of the calm economic winds, as a weather forecaster in Honolulu faces an easier task than one in Buffalo.
The other rationale you’ll sometimes hear for throwing out data is that there has been some sort of fundamental shift in the problem you are trying to solve. Sometimes these arguments are valid to a certain extent: the American economy is a constantly evolving thing and periodically undergoes structural shifts (recently, for instance, from an economy dominated by manufacturing to one dominated by its service sector). This isn’t baseball, where the game is always played by the same rules.
The problem with this is that you never know when the next paradigm shift will occur, and whether it will tend to make the economy more volatile or less so, stronger or weaker. An economic model conditioned on the notion that nothing major will change is a useless one. But anticipating these turning points is not easy.
Economic Data Is Very Noisy
The third major challenge for economic forecasters is that their raw data isn’t much good. I mentioned earlier that economic forecasters rarely provide their prediction intervals when they produce their forecasts—probably because doing so would undermine the public’s confidence in their expertise. “Why do people not give intervals? Because they’re embarrassed,” Hatzius says. “I think that’s the reason. People are embarrassed.”
The uncertainty, however, applies not just to economic forecasts but also to the economic variables themselves. Most economic data series are subject to revision, a process that can go on for months and even years after the statistics are first published. The revisions are sometimes enormous.46 One somewhat infamous example was the government’s estimate of GDP growth in the last quarter of 2008. Initially reported as “only” a 3.8 percent rate of decline, the economy is now believed to have been declining at almost 9 percent. Had they known the real size of the economic hole, the White House’s economists might have pushed for a larger stimulus package in January 2009, or they might have realized how deep the problems were and promoted a longer-term solution rather than attempting a quick fix.
Large errors like these have been fairly common. Between 1965 and 2009,47 the government’s initial estimates of quarterly GDP were eventually revised, on average, by 1.7 points. That is the average change; the range of possible changes in each quarterly GDP is higher still, and the margin of error48 on an initial quarterly GDP estimate is plus or minus 4.3 percent. That means there’s a chance that the economy will turn out to have been in recession even if the government had initially reported above-average growth, or vice versa. The government first reported that the economy had grown by 4.2 percent in the fourth quarter of 1977, for instance, but that figure was later revised to negative 0.1 percent.49
So we should have some sympathy for economic forecasters.50 It’s hard enough to know where the economy is going. But it’s much, much harder if you don’t know where it is to begin with.
A Butterfly Flaps Its Wings in Brazil and Someone Loses a Job in Texas
The challenge to economists might be compared to the one faced by weather forecasters. They face two of the same fundamental problems.
First, the economy, like the atmosphere, is a dynamic system: everything affects everything else and the systems are perpetually in motion. In meteorology, this problem is quite literal, since the weather is subject to chaos theory—a butterfly flapping its wings in Brazil can theoretically cause a tornado in Texas. But in loosely the same way, a tsunami in Japan or a longshoreman’s strike in Long Beach can affect whether someone in Texas finds a job.
Second, weather forecasts are subject to uncertain initial conditions. The probabilistic expression of weather forecasts (“there’s a 70 percent chance of rain”) arises not because there is any inherent randomness in the weather. Rather, the problem is that meteorologists assume they have imprecise measurements of what the initial conditions were like, and weather patterns (because they are subject to chaos theory) are extremely sensitive to changes in the initial conditions. In economic forecasting, likewise, the quality of the initial data is frequently quite poor.
Weather prediction, however, is one of the real success stories in this book. Forecasts of everything from hurricane trajectories to daytime high temperatures have gotten much better than they were even ten or twenty years ago, thanks to a combination of improved computer power, better data-collection methods, and old-fashioned hard work.
The same cannot be said for economic forecasting. Any illusion that economic forecasts were getting better ought to have been shattered by the terrible mistakes economists made in advance of the recent financial crisis.51
If the meteorologist shares some of the economist’s problems of a dynamic system with uncertain initial conditions, she has a wealth of hard science to make up for it. The physics and chemistry of something like a tornado are not all that complicated. That does not mean that tornadoes are easy to predict. But meteorologists have a strong fundamental understanding of what causes tornadoes to form and what causes them to dissipate.
Economics is a much softer science. Although economists have a reasonably sound understanding of the basic systems that govern the economy, the cause and effect are all blurred together, especially during bubbles and panics when the system is flushed with feedback loops contingent on human behavior.
Nevertheless, if discerning cause and effect is difficult for economists, it is probably better to try than just give up. Consider again, for instance, what Hatzius wrote on November 15, 2007:
The likely mortgage credit losses pose a significantly bigger macroeconomic risk than generally recognized. . . . The macroeconomic consequences could be quite dramatic. If leveraged investors see $200 [billion in] aggregate credit loss, they might need to scale back their lending by $2 trillion. This is a large shock. . . . It is easy to see how such a shock could produce a substantial recession or a long period of very sluggish g
rowth.
Consumers had been extended too much credit, Hatzius wrote, to pay for homes that the housing bubble had made unaffordable. Many of them had stopped making their mortgage payments, and there were likely to be substantial losses from this. The degree of leverage in the system would compound the problem, paralyzing the credit market and the financial industry more broadly. The shock might be large enough to trigger a severe recession.
And this is exactly how the financial crisis played out. Not only was Hatzius’s forecast correct, but it was also right for the right reasons, explaining the causes of the collapse and anticipating the effects. Hatzius refers to this chain of cause and effect as a “story.” It is a story about the economy—and although it might be a data-driven story, it is one grounded in the real world.
In contrast, if you just look at the economy as a series of variables and equations without any underlying structure, you are almost certain to mistake noise for a signal and may delude yourself (and gullible investors) into thinking you are making good forecasts when you are not. Consider what happened to one of Hatzius’s competitors, the forecasting firm ECRI.
In September 2011, ECRI predicted a near certainty of a “double dip” recession. “There’s nothing that policy makers can do to head it off,” it advised.52 “If you think this is a bad economy, you haven’t seen anything yet.” In interviews, the managing director of the firm, Lakshman Achuthan, suggested the recession would begin almost immediately if it hadn’t started already.53 The firm described the reasons for its prediction in this way: