In 1999, people said: ''internet businesses are doing great. "
Often we see only the businesses that do well and ignore the failures. Especially
179
in bull markets where successes get wide publicity. Ask: What is the relevant comparison population from which we measure 5-year performance (assume 100 Internet businesses)? How many are doing well (assume 5). How many are not performing well (assume 80). How many have gone out of business (assume 15). From this we can draw the conclusion that the above statement is false.
TransCorps technical department developed a new defense system and claims a success rate of80%. WhenJohn observed IO tests, he witnessed 8 failures and only 2 successes. In 1992, Theodore Postal, a professor of science and national security at MIT, measured the effectiveness (not in terms of its psychological and political impact) of the Patriot anti-missile system in the Gulf War. Based on studying videotapes of 26 Patriot/Scud engagements involving 25 misses and 1 hit, he told the Committee on Government Operations that, "the video evidence makes an overwhelming circumstantial case that Patriot did not come dose to achieving a 80 percent intercept rate in Saudi Arabia." The Pentagon reported firing 47 Patriot missiles at Scuds, at first claiming an 80% success rate. A congressional report later concluded that Patriots succeeded in downing only 4 Scuds.
If we assume that John's observations represent a valid random sample of tests,
how likely is it that he would observe exactly 2 "hits" and 8 "misses" if the technical department's claim was true?
This is an example of a binomial experiment. It describes experiments with repeated, identical trials where each trial can only have two possible outcomes (e.g. success or failure). Assuming independent engagements, the probability is only 0.007%. If the departments claim was true, then John witnessed an event of a very low probability. It is more likely that their claim was false.
Variability
You can, for example, never foretell what any man will do, but you can say with precision what an average number will be up to. Individuals vary, but percentages remain constant. So says the statistician.
- Arthur Conan Doyle (from The Sign of Four)
When evaluating likely outcomes, look at the whole distribution of possible outcomes
- average outcome, variability, and the probability of an extreme outcome and its consequences. What we mean by variability is how much the individual outcomes are spread out from the average outcome. The more spread out, the more variability. Ask: What has happened in the past? How much do the outcomes fluctuate around the average? What factors contribute to past variability? Have they changed?
180
The normal distribution curve shows a frequency distribution of outcomes and may sometimes help us find the most frequent outcomes and the variations. There are many ways a set of outcomes can be spread out or distributed. But some outcomes are more frequent than others. Many characteristics resulting from independent random factors have a bell-shaped frequency distribution. This means that the most frequent outcomes will be in the middle of the distribution, and the other outcomes will fall on either side of the middle. The
further away any outcome is from the middle, the less frequent it is.
Examples of normally distributed outcomes are heights or weights of adults (which depend on factors such as genes, diet, or environment), temperatures, car accidents, mortality rates, or the length of life of a light bulb.
What is the average height of female adults?
If we randomly select one thousand adult females and measure their height, we end up with a distribution of outcomes that look like a bell-shaped curve. Their heights will center around their average height, and the breadth of the curve indicates how variable around this average the heights are. Adding one extremely tall woman to a large sample doesn't really change the average height.
The same reasoning can be applied to an auto insurance company. The more of the same game the insurance company plays, the better the average becomes as a guide. Adding an extremely expensive car accident doesn't really change this. The average is representative and has predictive power.
But we can't use the normal distribution curve for those classes of insurance that involve mammoth and unusual risks. For an insurance company specializing in insuring unique events, the possible variability in outcomes is key.
TransCorp's new computer software grabbed 90% of the market.
In many cases, the normal distribution curve may not give us a true picture of reality. One single favorable or unfavorable extreme event can have a large impact and dramatically change the averages. We saw examples of this uneven type of distribution earlier with size and frequency. The less number of times or the shorter the time or the more impact a single event may have on the average, the more important it is to consider the variability, and the more unpredictable some factor becomes. For example, suppose the average sales of books are
$200,000. But if our sample contains one extreme best-seller (the Harry Potter
books), the average will not help us predict what a new book on average may sell for. Other examples are movie sales (the movie Titanic), price changes in financial markets (a sudden large depreciation of a currency, damages from
181
hurricanes (Katrina), or impact from innovations (TransCorp's new software). For example, there is no typical software and past averages don't mean anything. We see what has happened in the measured past. We don't know what the biggest hurricane will be or its likelihood of occurring. When we look back, we
only see what the biggest hurricane was in the documented past.
We can't use past statistics to predict these rare and high-impact events. We don't know their timing, frequency, or degree of impact. We can't exactly figure out their properties or develop a formula. We only know that they happen and that they can have a huge impact. In some cases, we may have some evidence telling us if the probabilities are changing. Also, many events are not independent but are instead interconnected. Financial markets are one example; markets have memories, assets may be correlated and one bad event may cause another. The danger of using the normal distribution curve in cases where one huge event can dramatically change reality is well described at length by the Epistemologist of Randomness Nassim Nicholas Taleb in his book The Black Swan.
10 people have a total wealth of$10 million i.e. the average wealth per person is $1 million.
What if one person has $10 million and nine nothing? How can the average
income soar? Suppose ten middle-class people are riding on a bus. One gets off the bus and a billionaire gets on.
John has an option to play one of two games. Each game has three equally likely outcomes. The game may be one of chance, insurance, investing, etc.
Outcomes Average outcome
Probability
Expected value
Gamel
30 40 50
40
1/3 1/3 1/3
40
Game2
0 20 100
40
1/3 1/3 1/3
40
If he plays each game over and over, the expected value is 40 for both, so it shouldn't matter what game he chooses to play. But what about ifJohn only plays this game once? He should play game 1 since it has less variability.
john has been offered to invest in a private venture capital fond. The venture manager's track record is an average rate of return of 25% over the last five years.
This doesn't say much if we don't look at how the underlying performance was produced. By looking closely at how this return was produced John found that the venture manager had done 10 deals. One deal had been a spectacular success
182
and the rest failures. Had this one deal been due to luck?
Remember that some people leave out data when reporting their performance.
Mary reads in the paper that the average price of a house is $1,000,000.
But this may cause her to
get a false picture of reality. Assume there are 100 houses and 90 of them are priced at $500,000 and 10 "castles" at $5.5 million. We have to watch out for the variations.
A business executive tells us that his company had average earnings of $50 million over the last 3 years. But when we look closer we find great variability and a downward trend in the performance record: 1998: $100 million, 1999: $50 million, 2000: $0.
The median is the middle of a distribution where half the outcomes are above the median and half below. If 9 people have $1 million each and one has $1 billion, the average wealth is about $101 million, but the median amount is $1 million.
In 1982, Stephen jay Gould was diagnosed with a rare and deadly form of cancer with a median mortality of 8 months after discovery. He beat the cancer for 20 years. Does an 8 month median mortality mean that a person will be dead in 8 months? Gould learned that an 8 month median mortality means half the people die before 8 months and half will live longer. But he found that there was a wider spread of outcomes after 8 months than before. This makes sense since there is a lower limit to the spread of outcomes during an 8 month period than after 8 months. No person will die before the start of the 8 month period but those who survive can live considerably longer than 8 months. We can't treat the average or the median as the most likely outcome for any single individual. Look at the variation between all outcomes. This means that a treatment ought to be determined based on whether an individual is likely to have an outcome better or worse than the median.
Effects of regression
"Regression to the mean" is a notion worked out by Sir Francis Galton (Charles Darwin's first cousin). It says that, in any series of events where chance is involved, very good or bad performances, high or low scores, extreme events, etc. tend on the average, to be followed by more average performance or less extreme events. If we do extremely well, we're likely to do worse the next time, while if we do poorly, we're likely to do better the next time. But regression to the mean is not a natural law. Merely a statistical tendency. And it may take a long time before it happens.
183
Dissatisfied with the new employees' performance, John put them into a skill enhancing program. He measures the employees' skills at the end of the program.
Their scores are now higher than they were on the first test. John's conclusion: "The skill-enhancing program caused the improvement in skill." This isn't necessarily true. Their higher scores could be the result of regression to the mean. Since these individuals were measured as being on the low end of the scale of skill, they would have shown an improvement even if they hadn't taken the skill-enhancing program. And there could be many reasons for their earlier performance - stress, fatigue, sickness, distraction, etc. Their true ability perhaps hasn't changed.
Our performance always varies around some average true performance. Extreme performance tends to get less extreme the next time. Why? Testing measurements can never be exact. All measurements are made up of one true part and one random error part. When the measurements are extreme, they are likely to be partly caused by chance. Chance is likely to contribute less on the second time we measure performance.
If we switch from one way of doing something to another merely because we
are unsuccessful, it's very likely that we do better the next time even if the new way of doing something is equal or worse.
Part Two and Three dealt with reasons for misjudgments and ideas for reducing them. Before we enter Part Four let's conclude with how we can learn from past mistakes.
Post Mortem
Spanish-American philosopher George Santayana once said: "Those who cannot remember the past are condemned to repeat it." How can we understand what is happening to us without any reference to the past? We conveniently forget to record our mistakes. But they should be highlighted. We should confess our errors and learn from them. We should look into their causes and take steps to prevent them from happening again. Ask:
What was my original reason for doing something? What did I know and what were my assumptions? What were my alternatives at the time?
How did reality work out relative to my original guess? What worked and what didn't?
Given the information that was available, should I have been able to predict what was going to happen?
What worked well? What should I do differently? What did I fail to do? What did I miss? What must I learn? What must I stop doing?
184
Why don't we do post mortems? Charles Munger says:
You tend to forget your own mistakes when reputation is threatened by remembering. For that very reason, one very wise company - Johnson & Johnson - has a system whereby two years or so after they've made some big acquisition they have a post-mortem. And they bring back the original projections and the original reasons for doing the deal. They identify the people who made the arguments and what have you. Then they compare them with how the deal worked out.
Warren Buffett says that, "Triumphs are trumpeted, but dumb decisions either get no follow-up or are rationalized." He continues:
Managers tend to be reluctant to look at the results of the capital projects or the acquisitions that they proposed with great detail only a year or two earlier to a board. And they don't want to actually stick the figures up there as to how the reality worked out relative to the projections. That's human nature.
But I think you're a better doctor if you drop by the pathology department occasionally. And I think you're a better manager or investor if you look at each decision that you've made of importance and see which ones worked out and which ones didn't - and figure your batting average. Then, if your batting average gets too bad, you better hand the decision-making over to someone else.
We could also use pre-mortems to help us anticipate problems and key vulnerabilities. For example, before making an important decision, imagine a failure where things really have gone wrong and ask: What could have caused this?
What guidelines and tools are there to better thinking? Charles Munger gives us some introductory remarks for Part Four:
Berkshire is basically a very old-fashioned kind of a place and we try to exert discipline to stay that way. I don't mean old-fashioned stupid. I mean the eternal verities: basic mathematics, basic horse sense, basic fear, basic diagnosis of human nature making possible predictions regarding human behavior. If you just do that with a certain amount of discipline, I think it's likely to work out quite well.
185
- PART FOUR -
GUIDELINES TO BETTER THINKING
The brain can be developed just the same way as the muscles can be developed, if one will only take the pains to train the mind to think.
- Thomas Alva Edison (American inventor, 1847-1931)
187
- ONE -
Sun Tzu said in The Art of War: "The general who wins a battle makes many calculations in his temple before the battle is fought."
The purpose of this part is to explore tools that provide a foundation for rational thinking. Ideas that help us when achieving goals, explaining "why," preventing and reducing mistakes, solving problems, and evaluating statements.
The following 12 tools will be discussed.
Models of reality.
Meaning.
Simplification.
Rules and filters.
Goals.
Alternatives.
Consequences.
Quantification.
Evidence.
Backward thinking.
Risk.
Attitudes.
MODELS OF REALITY
Educated men are as superior to uneducated men as the living are to the dead.
-Aristotle
Learn, understand and use the big ideas and general principles that explain a lot about how the world works. When Charles Munger was asked what would be the best question he should ask himself, he said:
Ifyou ask not about investment matters, but about your personal lives, I think the best question is, "Is the
re anything I can do to make my whole life and my whole mental process work better?"
189
And I would say that developing the habit of mastering the multiple models which underlie reality is the best thing you can do.. .It's just so much fun - and it works so well.
A model is an idea that helps us better understand how the world works. Models illustrate consequences and answer questions like "why" and "how". Take the model of social proof as an example. What happens? When people are uncertain they often automatically do what others do without thinking about the correct thing to do. This idea helps explain "why" and predict "how" people are likely to behave in certain situations.
Models help us avoid problems. Assume that we are told that the earth consists of infinite resources. By knowing the idea about limits, we know the statement is false. Someone gives us an investment proposal about a project that contradicts the laws of physics. How much misery can be avoided by staying away from whatever doesn't make scientific sense?
Seeking Wisdom Page 26