Likewise, when we look to the past, we do not feel any confusion about what we mean by the “events” that happened, nor does it seem difficult to say which of these events were important. And just as the uniqueness of the past causes us to think of the future as unique as well, so too does the apparent obviousness of past events tempt us into thinking that we ought to be able to anticipate which events will be important in the future. Yet what these commonsense notions overlook is that this view of the past is a product of a collective storytelling effort—not only by professional historians but also by journalists, experts, political leaders, and other shapers of public opinion—the goal of which is to make sense of “what happened.” Only once this story has been completed and agreed upon can we say what the relevant events were, or which were the most important. Thus it follows that predicting the importance of events requires predicting not just the events themselves but also the outcome of the social process that makes sense of them.
FROM COMMON SENSE TO UNCOMMON SENSE
For the purpose of going about our everyday business, none of this confusion may cause us serious problems. As I argued earlier, common sense is extraordinarily good at navigating particular circumstances. And because everyday decisions and circumstances are effectively broken up into many small chunks, each of which we get to deal with separately, it does not matter much that the sprawling hodgepodge of rules, facts, perceptions, beliefs, and instincts on which common sense relies forms a coherent whole. For the same reason, it may not matter much that commonsense reasoning leads to us think that we have understood the cause of something when in fact we have only described it, or to believe that we can make predictions that in fact we cannot make. By the time the future has arrived we have already forgotten most of the predictions we might have made about it, and so are untroubled by the possibility that most of them might have been wrong, or simply irrelevant. And by the time we get around to making sense of what did happen, history has already buried most of the inconvenient facts, freeing us to tell stories about whatever is left. In this way, we can skip from day to day and observation to observation, perpetually replacing the chaos of reality with the soothing fiction of our explanations. And for everyday purposes, that’s good enough, because the mistakes that we inevitably make don’t generally have any important consequences.
Where these mistakes do start to have important consequences is when we rely on our common sense to make the kinds of plans that underpin government policy or corporate strategy or marketing campaigns. By their very nature, foreign policy or economic development plans affect large numbers of people over extended periods of time, and so do need to work consistently across many different specific contexts. By their very nature, effective marketing or public health plans do depend on being able to reliably associate cause and effect, and so do need to differentiate scientific explanation from mere storytelling. By their very nature, strategic plans, whether for corporations or political parties, do necessarily make predictions about the future, and so do need to differentiate predictions that can be made reliably from those that cannot. And finally, all these sorts of plans do often have consequences of sufficient magnitude—whether financial, or political, or social—that it is worth asking whether or not there is a better, uncommonsense way to go about making them. It is therefore to the virtues of uncommon sense, and its implications for prediction, planning, social justice, and even social science, that we now turn.
PART TWO
UNCOMMON SENSE
CHAPTER 7
The Best-Laid Plans
The message of the previous chapter is that the kinds of predictions that common sense tells us we ought to be able to make are in fact impossible—for two reasons. First, common sense tells us that only one future will actually play out, and so it is natural to want to make specific predictions about it. In complex systems, however, which comprise most of our social and economic life, the best we can hope for is to reliably estimate the probabilities with which certain kinds of events will occur. Second, common sense also demands that we ignore the many uninteresting, unimportant predictions that we could be making all the time, and focus on those outcomes that actually matter. In reality, however, there is no way to anticipate, even in principle, which events will be important in the future. Even worse, the black swan events that we most wish we could have predicted are not really events at all, but rather shorthand descriptions—“the French Revolution,” “the Internet,” “Hurricane Katrina,” “the global financial crisis”—of what are in reality whole swaths of history. Predicting black swans is therefore doubly hopeless, because until history has played out it’s impossible even to know what the relevant terms are.
It’s a sobering message. But just because we can’t make the kinds of predictions we’d like to make doesn’t mean that we can’t predict anything at all. As any good poker player can tell you, counting cards won’t tell you exactly which card is going to show up next, but by knowing the odds better than your opponents you can still make a lot of money over time by placing more informed bets, and winning more often than you lose.1 And even for outcomes that truly can’t be predicted with any reliability whatsoever, just knowing the limits of what’s possible can still be helpful—because it forces us to change the way we plan. So what kinds of predictions can we make, and how can we make them as accurately as possible? And how should we change the way we think about planning—in politics, business, policy, marketing, and management—to accommodate the understanding that some predictions cannot be made at all? These questions may seem distant from the kinds of issues and puzzles that we grapple with on an everyday basis, but one way or another—through their influence on the firms we work for, or the economy at large, or the issues that we read about every day in the newspaper—they affect us all.
WHAT CAN WE PREDICT?
To oversimplify somewhat, there are two kinds of events that arise in complex social systems—events that conform to some stable historical pattern, and events that do not—and it is only the first kind about which we can make reliable predictions. As I discussed in the previous chapter, even for these events we can’t predict any particular outcome any more than we can predict the outcome of any particular die roll. But as long as we can gather enough data on their past behavior, we can do a reasonable job of predicting probabilities, and that can be enough for many purposes.
Every year, for example, each of us may or may not be unlucky enough to catch the flu. The best anyone can predict is that in any given season we would have some probability of getting sick. Because there are so many of us, however, and because seasonal influenza trends are relatively consistent from year to year, drug companies can do a reasonable job of anticipating how many flu shots they will need to ship to a given part of the world in a given month. Likewise, consumers with identical financial backgrounds may vary widely in their likelihood of defaulting on a credit card, depending on what is going on in their lives. But credit card companies can do a surprisingly good job of predicting aggregate default rates by paying attention to a range of socioeconomic, demographic, and behavioral variables. And Internet companies are increasingly taking advantage of the mountains of Web-browsing data generated by their users to predict the probability that a given user will click on a given search result, respond favorably to particular news story, or be swayed by a particular recommendation. As the political scientist Ian Ayres writes in his book Super Crunchers, predictions of this kind are being made increasingly in highly data-intensive industries like finance, healthcare, and e-commerce, where the often modest gains associated with data-driven predictions can add up over millions or even billions of tiny decisions—in some cases every day—to produce very substantial gains to the bottom line.2
So far, so good. But there are also many areas of business—as well as of government and policy—that rely on predictions that do not quite fit into this supercrunching mold. For example, whenever a book publisher decides how much of an advance to offer a potentia
l author, it is effectively making a prediction about the future sales of the proposed book. The more copies the book sells, the more royalties the author is entitled to, and so the more of an advance the publisher should offer to prevent the author from signing with a different publisher. But if in making this calculation, the publisher overestimates how well the book will sell, it will end up overpaying the author—good for the author but bad for the publisher’s bottom line. Likewise when a movie studio decides to green-light a project, it is effectively making a prediction about the future revenues of the movie, and thus how much it can afford to spend making and marketing it. Or when a drug company decides to proceed with the clinical testing stage of a new drug, it must justify the enormous expense in terms of some prediction about the likely success of the trial and the eventual market size for the drug.
All these lines of business therefore depend on predictions, but they are considerably more complicated predictions than predictions about the number of flu cases expected in North America this winter, or the probability that a given user will click on a given ad online. When a publisher offers an advance for a book, the book itself is typically at least a year or two away from publication; so the publisher has to make a prediction not only about how the book itself will turn out but also what the market will be like for that kind of book when it is eventually published, how it will be reviewed, and any number of other related factors. Likewise predictions about movies, new drugs, and other kinds of business or development projects are, in effect, predictions about complex, multifaceted processes that play out over months or years. Even worse, because decision makers are constrained to making only a handful of such decisions every year, they do not have the luxury of averaging out their uncertainty over huge numbers of predictions.
Nevertheless, even in these cases, decision makers often have at least some historical data on which to draw. Publishers can keep track of how many copies they have sold of similar books in the past, while movie studios can do the same for box office revenues, DVD sales, and merchandising profits. Likewise, drug companies can assess the rates with which similar drugs have succeeded in reaching the market, marketers can track the historical success of comparable products, and magazine publishers can track the newsstand sales of previous cover stories. Decision makers often also have a lot of other data on which to draw—including market research, internal evaluations of the project in question, and their knowledge of the industry in general. So as long as nothing dramatic changes in the world between when they commit to a project and when it launches, then they are still in the realm of predictions that are at least possible to make reliably. How should they go about making them?
MARKETS, CROWDS, AND MODELS
One increasingly popular method is to use what is called a prediction market—meaning a market in which buyers and sellers can trade specially designed securities whose prices correspond to the predicted probability that a specific outcome will take place. For example, the day before the 2008 US presidential election, an investor could have paid $0.92 for a contract in the Iowa Electronic Markets—one of the longest-running and best-known prediction markets—that would have yielded him or her $1 if Barack Obama had won. Participants in prediction markets therefore behave much like participants in financial markets, buying and selling contracts for whatever price is on offer. But in the case of prediction markets, the prices are explicitly interpreted as making a prediction about the outcome in question—for example, the probability of an Obama victory on the eve of Election Day was predicted by the Iowa Electronic Markets to be 92 percent.
In generating predictions like this one, prediction markets exploit a phenomenon that New Yorker writer James Surowiecki dubbed the “wisdom of crowds”—the notion that although individual people tend to make highly error-prone predictions, when lots of these estimates are averaged together, the errors have a tendency to cancel out; hence the market is in some sense “smarter” than its constitutents. Many such markets also require participants to bet real money, thus people who know something about a particular topic are more likely to participate than people who don’t. What’s so powerful about this feature of prediction markets is that it doesn’t matter who has the relevant market information—a single expert or a large number of nonexperts, or any combination in between. In theory, the market should incorporate all their opinions in proportion to how much each is willing to bet. In theory, in fact, no one should be able to consistently outperform a properly designed prediction market. The reason is that if someone could outperform the market, they would have an incentive to make money in it. But the very act of making money in the market would immediately shift the prices to incorporate the new information.3
The potential of prediction markets to tap into collective wisdom has generated a tremendous amount of excitement among professional economists and policy makers alike. Imagine, for example, that a market had been set up to predict the possibility of a catastrophic failure in deep-water oil drilling in the Gulf prior to the BP disaster in April 2010. Possibly insiders like BP engineers could have participated in the market, effectively making public what they knew about the risks their firms were taking. Possibly then regulators would have had a more accurate assessment of those risks and been more inclined to crack down on the oil industry before a disaster took place. Possibly the disaster could have been averted. These are the sorts of claims that the proponents of prediction markets tend to make, and it’s easy to see why they’ve generated so much interest. In recent years, in fact, prediction markets have been set up to make predictions as varied as the likely success of new products, the box office revenues of upcoming movies, and the outcomes of sporting events.
In practice, however, prediction markets are more complicated than the theory suggests. In the 2008 presidential election, for example, one of the most popular prediction markets, Intrade, experienced a series of strange fluctuations when an unknown trader started placing very large bets on John McCain, generating large spikes in the market’s prediction for a McCain victory. Nobody figured out who was behind these bets, but the suspicion was that it was a McCain supporter or even a member of the campaign. By manipulating the market prices, he or she was trying to create the impression that a respected source of election forecasts was calling the election for McCain, presumably with the hope of creating a self-fulfilling prophecy. It didn’t work. The spikes were quickly reversed by other traders, and the mystery bettor ended up losing money; thus the market functioned essentially as it was supposed to. Nevertheless, it exposed a potential vulnerability of the theory, which assumes that rational traders will not deliberately lose money. The problem is that if the goal of a participant is instead to manipulate perceptions of people outside the market (like the media) and if the amounts involved are relatively small (tens of thousands of dollars, say, compared with the tens of millions of dollars spent on TV advertising), then they may not care about losing money, in which case it’s no longer clear what signal the market is sending.4
Problems like this one have led some skeptics to claim that prediction markets are not necessarily superior to other less sophisticated methods, such as opinion polls, that are harder to manipulate in practice. However, little attention has been paid to evaluating the relative performance of different methods, so nobody really knows for sure.5 To try to settle the matter, my colleagues at Yahoo! Research and I conducted a systematic comparison of several different prediction methods, where the predictions in question were the outcomes of NFL football games. To begin with, for each of the fourteen to sixteen games taking place each weekend over the course of the 2008 season, we conducted a poll in which we asked respondents to state the probability that the home team would win as well as their confidence in their prediction. We also collected similar data from the website Probability Sports, an online contest where participants can win cash prizes by predicting the outcomes of sporting events. Next, we compared the performance of these two polls with the Vegas sports betting market�
�one of the oldest and most popular betting markets in the world—as well as with another prediction market, TradeSports. And finally, we compared the prediction of both the markets and the polls against two simple statistical models. The first model relied only on the historical probability that home teams win—which they do 58 percent of the time—while the second model also factored in the recent win-loss records of the two teams in question. In this way, we set up a six-way comparison between different prediction methods—two statistical models, two markets, and two polls.6
Given how different these methods were, what we found was surprising: All of them performed about the same. To be fair, the two prediction markets performed a little better than the other methods, which is consistent with the theoretical argument above. But the very best performing method—the Las Vegas Market—was only about 3 percentage points more accurate than the worst-performing method, which was the model that always predicted the home team would win with 58 percent probability. All the other methods were somewhere in between. In fact, the model that also included recent win-loss records was so close to the Vegas market that if you used both methods to predict the actual point differences between the teams, the average error in their predictions would differ by less than a tenth of a point. Now, if you’re betting on the outcomes of hundreds or thousands of games, these tiny differences may still be the difference between making and losing money. At the same time, however, it’s surprising that the aggregated wisdom of thousands of market participants, who collectively devote countless hours to analyzing upcoming games for any shred of useful information, is only incrementally better than a simple statistical model that relies only on historical averages.
Everything Is Obvious Page 16