At the end of the first year, we still expect to have 4,500 managers turning a profit (45% of them), the second, 45% of that number, 2,025. The third, 911; the fourth, 410; the fifth, 184. Let us give the surviving managers names and dress them in business suits. True, they represent less than 2% of the original cohort. But they will get attention. Nobody will mention the other 98%.What can we conclude?
The first counterintuitive point is that a population entirely composed of bad managers will produce a small amount of great track records. As a matter of fact, assuming the manager shows up unsolicited at your door, it will be practically impossible to figure out whether he is good or bad. The results would not markedly change even if the population were composed entirely of managers who are expected in the long run to lose money. Why? Because owing to volatility, some of them will make money. We can see here that volatility actually helps bad investment decisions.
The second counterintuitive point is that the expectation of the maximum of track records, with which we are concerned, depends more on the size of the initial sample than on the individual odds per manager. In other words, the number of managers with great track records in a given market depends far more on the number of people who started in the investment business (in place of going to dental school), rather than on their ability to produce profits. It also depends on the volatility. Why do I use the notion of expectation of the maximum? Because I am not concerned at all with the average track record. I will get to see only the best of the managers, not all of the managers. This means that we would see more “excellent managers” in 2006 than in 1998, provided the cohort of beginners was greater in 2001 than it was in 1993—I can safely say that it was.
Regression to the Mean
The “hot hand in basketball” is another example of misperception of random sequences: It is very likely in a large sample of players for one of them to have an inordinately lengthy lucky streak. As a matter of fact it is very unlikely that an unspecified player somewhere does not have an inordinately lengthy lucky streak. This is a manifestation of the mechanism called regression to the mean. I can explain it as follows:
Generate a long series of coin flips producing heads and tails with 50% odds each and fill up sheets of paper. If the series is long enough you may get eight heads or eight tails in a row, perhaps even ten of each. Yet you know that in spite of these wins the conditional odds of getting a head or a tail is still 50%. Imagine these heads and tails as monetary bets filling up the coffers of an individual. The deviation from the norm as seen in excess heads or excess tails is here entirely attributable to luck, in other words, to variance, not to the skills of the hypothetical player (since there is an even probability of getting either).
A result is that in real life, the larger the deviation from the norm, the larger the probability of it coming from luck rather than skills: Consider that even if one has 55% probability of heads, the odds of ten wins is still very small. This can be easily verified in stories of very prominent people in trading rapidly reverting to obscurity, like the heroes I used to watch in trading rooms. This applies to height of individuals or the size of dogs. In the latter case, consider that two average-sized parents produce a large litter. The largest dogs, if they diverge too much from the average, will tend to produce offspring of smaller size than themselves, and vice versa. This “reversion” for the large outliers is what has been observed in history and explained as regression to the mean. Note that the larger the deviation, the more important its effect.
Again, one word of warning: All deviations do not come from this effect, but a disproportionately large proportion of them do.
Ergodicity
To get more technical, I have to say that people believe that they can figure out the properties of the distribution from the sample they are witnessing. When it comes to matters that depend on the maximum, it is altogether another distribution that is being inferred, that of the best performers. We call the difference between the average of such distribution and the unconditional distribution of winners and losers the survivorship bias—here the fact that about 3% of the initial cohort discussed earlier will make money five years in a row. In addition, this example illustrates the properties of ergodicity, namely, that time will eliminate the annoying effects of randomness. Looking forward, in spite of the fact that these managers were profitable in the past five years, we expect them to break even in any future time period. They will fare no better than those of the initial cohort who failed earlier in the exercise. Ah, the long term.
A few years ago, when I told one A., a then Master-of-the-Universe type, that track records were less relevant than he thought, he found the remark so offensive that he violently flung his cigarette lighter in my direction. The episode taught me a lot. Remember that nobody accepts randomness in his own success, only his failure. His ego was pumped up as he was heading up a department of “great traders” who were then temporarily making a fortune in the markets and attributing the idea to the soundness of their business, their insights, or their intelligence. They subsequently blew up during the harsh New York winter of 1994 (it was the bond market crash that followed the surprise interest rate hike by Alan Greenspan). The interesting part is that several years later I can hardly find any of them still trading (ergodicity).
Recall that the survivorship bias depends on the size of the initial population. The information that a person derived some profits in the past, just by itself, is neither meaningful nor relevant. We need to know the size of the population from which he came. In other words, without knowing how many managers out there have tried and failed, we will not be able to assess the validity of the track record. If the initial population includes ten managers, then I would give the performer half my savings without a blink. If the initial population is composed of 10,000 managers, I would ignore the results. The latter situation is generally the case; these days so many people have been drawn to the financial markets. Many college graduates are trading as a first career, failing, then going to dental school.
If, as in a fairy tale, these fictional managers materialized into real human beings, one of these could be the person I am meeting tomorrow at 11:45 a.m. Why did I select 11:45 a.m.? Because I will question him about his trading style. I need to know how he trades. I will then be able to claim that I have to rush to a lunch appointment if the manager puts too much emphasis on his track record.
LIFE IS COINCIDENTAL
Next, we look at the extensions to real life of our bias in the understanding of the distribution of coincidences.
The Mysterious Letter
You get an anonymous letter on January 2 informing you that the market will go up during the month. It proves to be true, but you disregard it owing to the well-known January effect (stocks have gone up historically during January). Then you receive another one on February 1 telling you that the market will go down. Again, it proves to be true. Then you get another letter on March 1—same story. By July you are intrigued by the prescience of the anonymous person and you are asked to invest in a special offshore fund. You pour all your savings into it. Two months later, your money is gone. You go spill your tears on your neighbor’s shoulder and he tells you that he remembers that he received two such mysterious letters. But the mailings stopped at the second letter. He recalls that the first one was correct in its prediction, the other incorrect.
What happened? The trick is as follows. The con operator pulls 10,000 names out of a phone book. He mails a bullish letter to one half of the sample, and a bearish one to the other half. The following month he selects the names of the persons to whom he mailed the letter whose prediction turned out to be right, that is, 5,000 names. The next month he does the same with the remaining 2,500 names, until the list narrows down to 500 people. Of these there will be 200 victims. An investment in a few thousand dollars’ worth of postage stamps will turn into several million.
An Interrupted Tennis Game
It is not uncommon for someone watching a
tennis game on television to be bombarded by advertisements for funds that did (until that minute) outperform others by some percentage over some period. But, again, why would anybody advertise if he didn’t happen to outperform the market? There is a high probability of the investment coming to you if its success is caused entirely by randomness. This phenomenon is what economists and insurance people call adverse selection. Judging an investment that comes to you requires more stringent standards than judging an investment you seek, owing to such selection bias. For example, by going to a cohort composed of 10,000 managers, I have 2/100 chances of finding a spurious survivor. By staying home and answering my doorbell, the chance of the soliciting party being a spurious survivor is closer to 100%.
Reverse Survivors
We have so far discussed the spurious survivor—the same logic applies to the skilled person who has the odds markedly stacked in her favor, but who still ends up going to the cemetery. This effect is the exact opposite of the survivorship bias. Consider that all one needs is two bad years in the investment industry to terminate a risk-taking career and that, even with great odds in one’s favor, such an outcome is very possible. What do people do to survive? They maximize their odds of staying in the game by taking black-swan risks (like John and Carlos)—those that fare well most of the time, but incur a risk of blowing up.
The Birthday Paradox
The most intuitive way to describe the data mining problem to a nonstatistician is through what is called the birthday paradox, though it is not really a paradox, simply a perceptional oddity. If you meet someone randomly, there is a one in 365.25 chance of your sharing their birthday, and a considerably smaller one of having the exact birthday of the same year. So, sharing the same birthday would be a coincidental event that you would discuss at the dinner table. Now let us look at a situation where there are 23 people in a room. What is the chance of there being 2 people with the same birthday? About 50%. For we are not specifying which people need to share a birthday; any pair works.
It’s a Small World!
A similar misconception of probabilities arises from the random encounters one may have with relatives or friends in highly unexpected places. “It’s a small world!” is often uttered with surprise. But these are not improbable occurrences—the world is much larger than we think. It is just that we are not truly testing for the odds of having an encounter with one specific person, in a specific location at a specific time. Rather, we are simply testing for any encounter, with any person we have ever met in the past, and in any place we will visit during the period concerned. The probability of the latter is considerably higher, perhaps several thousand times the magnitude of the former.
When the statistician looks at the data to test a given relationship, say, to ferret out the correlation between the occurrence of a given event, like a political announcement, and stock market volatility, odds are that the results can be taken seriously. But when one throws the computer at data, looking for just about any relationship, it is certain that a spurious connection will emerge, such as the fate of the stock market being linked to the length of women’s skirts. And just like the birthday coincidences, it will amaze people.
Data Mining, Statistics, and Charlatanism
What is your probability of winning the New Jersey lottery twice? One in 17 trillion. Yet it happened to Evelyn Adams, whom the reader might guess should feel particularly chosen by destiny. Using the method we developed above, researchers Percy Diaconis and Frederick Mosteller estimated at 30 to 1 the probability that someone, somewhere, in a totally unspecified way, gets so lucky!
Some people carry their data mining activities into theology—after all, ancient Mediterraneans used to read potent messages in the entrails of birds. An interesting extension of data mining into biblical exegesis is provided in The Bible Code by Michael Drosnin. Drosnin, a former journalist (seemingly innocent of any training in statistics), aided by the works of a “mathematician,” helped “predict” the former Israeli Prime Minister Yitzhak Rabin’s assassination by deciphering a bible code. He informed Rabin, who obviously did not take it too seriously. The Bible Code finds statistical irregularities in the Bible; these help predict some such events. Needless to say that the book sold well enough to warrant a sequel predicting with hindsight even more such events.
The same mechanism is behind the formation of conspiracy theories. Like The Bible Code they can seem perfect in their logic and can cause otherwise intelligent people to fall for them. I can create a conspiracy theory by downloading hundreds of paintings from an artist or group of artists and finding a constant among all those paintings (among the hundreds of thousand of traits). I would then concoct a conspiratorial theory around a secret message shared by these paintings. This is seemingly what the author of the bestselling The Da Vinci Code did.
The Best Book I Have Ever Read!
My favorite time is spent in bookstores, where I aimlessly move from book to book in an attempt to make a decision as to whether to invest the time in reading it. My buying is frequently made on impulse, based on superficial but suggestive clues. Frequently, I have nothing but a book jacket as appendage to my decision making. Jackets often contain praise by someone, famous or not, or excerpts from a book review. Good praise by a famous and respected person or a well-known magazine would sway me into buying the book.
What is the problem? I tend to confuse a book review, which is supposed to be an assessment of the quality of the book, with the best book reviews, marred with the same survivorship biases. I mistake the distribution of the maximum of a variable with that of the variable itself. The publisher will never put on the jacket of the book anything but the best praise. Some authors go even a step beyond, taking a tepid or even unfavorable book review and selecting words in it that appear to praise the book. One such example came from one Paul Wilmott (an English financial mathematician of rare brilliance and irreverence) who managed to announce that I gave him his “first bad review,” yet used excerpts from it as praise on the book jacket (we later became friends, which allowed me to extract an endorsement from him for this book).
The first time I was fooled by this bias was upon buying, when I was sixteen, Manhattan Transfer, a book by John Dos Passos, the American writer, based on praise on the jacket by the French writer and “philosopher” Jean-Paul Sartre, who claimed something to the effect that Dos Passos was the greatest writer of our time. This simple remark, possibly blurted out in a state of intoxication or extreme enthusiasm, caused Dos Passos to become required reading in European intellectual circles, as Sartre’s remark was mistaken for a consensus estimate of the quality of Dos Passos rather than what it was, the best remark. (In spite of such interest in his work, Dos Passos has reverted to obscurity.)
The Backtester
A programmer helped me build a backtester. It is a software program connected to a database of historical prices, which allows me to check the hypothetical past performance of any trading rule of average complexity. I can just apply a mechanical trading rule, like buy NASDAQ stocks if they close more than 1.83% above their average of the previous week, and immediately get an idea of its past performance. The screen will flash my hypothetical track record associated with the trading rule. If I do not like the results, I can change the percentage to, say, 1.2%. I can also make the rule more complex. I will keep trying until I find something that works well.
What am I doing? The exact same task of looking for the survivor within the set of rules that can possibly work. I am fitting the rule on the data. This activity is called data snooping. The more I try, the more I am likely, by mere luck, to find a rule that worked on past data. A random series will always present some detectable pattern. I am convinced that there exists a tradable security in the Western world that would be 100% correlated with the changes in temperature in Ulan Bator, Mongolia.
To get technical, there are even worse extensions. An outstanding recent paper by Sullivan, Timmerman, and White goes further and considers that
the rules that may be in use successfully today may be the result of a survivorship bias.
Suppose that, over time, investors have experimented with technical trading rules drawn from a very wide universe—in principle thousands of parameterizations of a variety of types of rules. As time progresses, the rules that happen to perform well historically receive more attention and are considered “serious contenders” by the investment community, while unsuccessful trading rules are more likely to be forgotten. . . . If enough trading rules are considered over time, some rules are bound by pure luck, even in a very large sample, to produce superior performance even if they do not genuinely possess predictive power over asset returns. Of course, inference based solely on the subset of surviving trading rules may be misleading in this context since it does not account for the full set of initial trading rules, most of which are unlikely to have underperformed.
I have to decry some excesses in backtesting that I have closely witnessed in my private career. There is an excellent product designed just for that, called Omega TradeStation, that is currently on the market, in use by tens of thousands of traders. It even offers its own computer language. Beset with insomnia, the computderized day traders become night testers plowing the data for some of its properties. By dint of throwing their monkeys on typewriters, without specifying what book they want their monkey to write, they will hit upon hypothetical gold somewhere. Many of them blindly believe in it.
Fooled by Randomness Page 18