In 1994, Benter published a paper outlining his basic betting model. He included a table that showed how his predictions compared to actual race outcomes. The results looked pretty good. Apart from a few discrepancies here and there, the model was remarkably close to reality. However, Benter warned that the results hid a major flaw. If anyone had tried to bet using the predictions, it would have been catastrophic.
SUPPOSE YOU CAME INTO some money and wanted to use the windfall to buy a little bookstore somewhere. There are a couple of ways you could go about it. Having drawn up a short list of stores you might buy, you could go into each one, check the inventory, quiz the management, and examine the accounts. Or you could bypass the paperwork and simply sit outside and count how many customers go in and how many books they come out with. These contrasting strategies reflect the two main ways people approach investing. Researching a company to its core is known as “fundamental analysis,” whereas watching how other people view the company over time is referred to as “technical analysis.”
Bolton and Chapman’s predictions used a fundamental approach. Such methods rely on having good information and sifting through it in the best way possible. The views of pundits don’t feature in the analysis. It doesn’t matter what other people are doing and which horses they are choosing. The model ignores the betting market. It’s like making predictions in a vacuum.
Although it might be possible to make predictions about races in isolation, the same cannot be said for betting on them. If syndicates want to make money at the track, they need to outwit other gamblers. This is where purely fundamental approaches can run into problems. When Benter compared the predictions from his fundamental model with the public odds, he noticed a worrying bias. He’d used the model to find “overlays”: horses that, according to the model, have a better chance of winning than their odds imply. These were the horses he would bet on if he were hoping to beat other gamblers. Yet when Benter looked at actual race results, the overlays did not win as often as the predictions suggested. It seemed that the true chances of these horses winning lay somewhere between the probability given by the model and the probability implied by the betting odds. The fundamental approach was clearly missing something.
Even if a betting team has a good model, the public’s views on a horse’s chances—as shown by the odds on the tote board—aren’t completely irrelevant, because not every gambler picks horses based on publicly available information. Some people might know about the jockey’s strategy for the race or the horse’s eating and workout schedule. When they try to capitalize on this privileged information, it changes the odds on the board.
It makes sense to combine the two available sources of expertise, namely, the model and opinion of other gamblers (as shown by the odds on the tote board). This is the approach Benter advocated. His model still ignores the public odds initially. The first set of predictions is made as if there is no such thing as betting. These predictions are then merged with the public’s view. The probability each horse will win is a balance between the chance of the horse winning in the model and the chance of victory according to the current odds. The scales can tip one way or the other: whichever produces the combined prediction that lines up best with actual results. Strike the right balance, and good predictions can become profitable ones.
WHEN WOODS AND BENTER arrived in Hong Kong, they did not meet with immediate success. While Benter spent the first year putting together the statistical model, Woods tried to make money exploiting the long-shot-favorite bias. They had come to Asia with a bankroll of $150,000; within two years, they’d lost it all. It didn’t help that investors weren’t interested in their strategy. “People had so little faith in the system that they would not have invested for 100 percent of the profits,” Woods later said.
By 1986, things were looking better. After writing hundreds of thousands of lines of computer code, Benter’s model was ready to go. The team had also collected enough race results to generate decent predictions. Using the model to select horses, they took home $100,000 that year.
Disagreements meant the partnership ended after that first successful season. Before long, Woods and Benter had formed rival syndicates and continued to compete against each other in Hong Kong. Although Woods later admitted that Benter’s team had the better model, both groups saw their profits rise dramatically over the next few years.
Several betting syndicates in Hong Kong now use models to predict horse races. Because the track takes a cut, it’s difficult to make money on simple bets such as picking the winner. Instead, syndicates chase the more complicated wagers on offer. These include the trifecta: to win, gamblers must predict the horses that will finish first, second, and third in correct order. Then there’s the triple trio, which involves winning three trifectas in a row. Although the payoffs for these exotic bets can be huge, the margin for error is also much smaller.
One of the drawbacks with Bolton and Chapman’s original model is that it assumes the same level of uncertainty for all the horses. This makes the calculations easier, but it means sacrificing some realism. To illustrate the problem, imagine two horses. The first is a bastion of reliability, always finishing the race in about the same time. The second is more variable, sometimes finishing much quicker than the first, but sometimes taking much longer. As a result, both horses take the same time on average to run a race.
If just these two horses are racing, they will have an equal probability of winning. It might as well be a coin toss. But what if several horses are in the race, each with a different level of uncertainty? If a betting team wants to pick the top three accurately, they need to account for these differences. For years, this was beyond the reach of even the best horse racing models. In the past decade, though, syndicates have found a way to predict races with a varying amount of uncertainty looming over each horse. It’s not just recent increases in computing power that have made this possible. The predictions also rely on a much older idea, originally developed by a group of mathematicians working on the hydrogen bomb.
ONE EVENING IN JANUARY 1946, Stanislaw Ulam went to bed with a terrible headache. When he woke up the next morning, he’d lost the ability to speak. He was rushed to a Los Angeles hospital, where concerned surgeons drilled a hole in his skull. Finding his brain severely inflamed as a result of infection, they treated the exposed tissue with penicillin to halt the disease.
Born in Poland, Ulam had left Europe for the United States only weeks before his country fell to the Nazis in September 1939. He was a mathematician by training and had spent most of the Second World War working on the atomic bomb at Los Alamos National Laboratory. After the conflict ended, Ulam joined UCLA as a professor of mathematics. It wasn’t his first choice: amid rumors that Los Alamos might close after the war, Ulam had applied to several higher-profile universities that all turned him down.
By Easter 1946, Ulam had fully recovered from his operation. The stay in the hospital had given him time to consider his options, and he decided to quit his job at UCLA and return to Los Alamos. Far from shutting it down, the government was now pouring money into the laboratory. Much of the effort was going into building a hydrogen bomb, nicknamed the “Super.” When Ulam arrived, several obstacles were still in the way. In particular, the researchers needed a means to predict the nuclear chain reactions involved in a detonation. This meant working out how often neutrons collide—and hence how much energy they would give off—inside a bomb. To Ulam’s frustration, this couldn’t be calculated using conventional mathematics.
Ulam did not enjoy grinding away at problems for hours, as many mathematicians spent their time doing. A colleague once recalled him trying to solve a quadratic equation on a blackboard. “He furrowed his brow in rapt absorption, while scribbling formulas in his tiny handwriting. When he finally got the answer, he turned around and said with relief, ‘I feel I have done my work for the day.’”
Ulam preferred to focus on creating new ideas; others could fill in the technical detail
s. It wasn’t just mathematical puzzles he tackled in inventive ways. While working at the University of Wisconsin during the winter of 1943, he’d noticed that several of his colleagues were no longer showing up to work. Soon afterward, Ulam received an invitation to join a project in New Mexico. The letter didn’t say what was involved. Intrigued, Ulam headed to the campus library and tried to find out all he could about New Mexico. It turned out that there was only one book about the state. Ulam looked at who’d checked it out recently. “Suddenly, I knew where all my friends had disappeared to,” he said. Glancing over the others’ research interests, he quickly pieced together what they were all working on out in the desert.
WITH HIS HYDROGEN BOMB calculations turning into a series of mathematical cul-de-sacs, Ulam remembered a puzzle he’d thought about during his stay in the hospital. While recovering from surgery, he had passed the time playing solitaire. During one game, he’d tried to work out the probability of a certain card arrangement appearing. Faced with having to calculate a vast set of possibilities—the sort of monotonous work he usually tried to avoid—Ulam realized it might be quicker just to lay out the cards several times and watch what happened. If he repeated the experiment enough times, he would end up with a good idea of the answer without doing a single calculation.
Wondering whether the same technique could also help with the neutron problem, Ulam took the idea to one of his closest colleagues, a mathematician by the name of John von Neumann. The two had known each other for over a decade. It was von Neumann who’d suggested Ulam leave Poland for America in the 1930s; he’d also been the one who invited Ulam to join Los Alamos in 1943. They made quite the pair, portly von Neumann in his immaculate suits—jacket always on—and Ulam with his absent-minded fashion sense and dazzling green eyes.
Von Neumann was quick-witted and logical, sometimes to the point of being blunt. He’d once grown hungry during a train journey and had asked the conductor to send the sandwich seller his way. The request fell on unsympathetic ears. “I will if I see him,” the conductor said. To which von Neumann replied, “This train is linear, isn’t it?”
When Ulam described his solitaire idea, von Neumann immediately spotted its potential. Enlisting the help of another colleague, a physicist named Nicholas Metropolis, they outlined a way to solve the chain reaction problem by repeatedly simulating neutron collisions. This was possible thanks to the recent construction of a programmable computer at Los Alamos. Because they worked for a government agency, however, the trio needed a code name for their new approach. With a nod to Ulam’s heavy-gambling uncle, Metropolis suggested they call it the “Monte Carlo method.”
Because the method involved repeated simulations of random events, the group needed access to lots of random numbers. Ulam joked that they should hire people to sit rolling dice all day. His flippancy hinted at an unfortunate truth: generating random numbers was a genuinely difficult task, and they needed a lot of them. Even if those nineteenth-century Monte Carlo journalists had been honest, Karl Pearson would have struggled to build a collection big enough for the Los Alamos men.
Von Neumann, inventive as ever, instead came up with a method for creating “pseudorandom” numbers using simple arithmetic. Despite its being easy to implement, von Neumann knew his method had shortcomings, chiefly the fact that it couldn’t generate truly random numbers. “Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin,” he later joked.
As computers have increased in power, and good pseudorandom numbers have become more readily available, the Monte Carlo method has become a valuable tool for scientists. Edward Thorp even used Monte Carlo simulations to produce the strategies in Beat the Dealer. However, things aren’t so straightforward in horse racing.
In blackjack, only so many combinations of cards can come up—too many to solve the game by hand, but not by computer. Compare this with horse racing models, which can have over a hundred factors. It’s possible to tweak the contribution of each one—and hence change the prediction—in a vast number of ways. By just randomly picking different contributions, it’s very unlikely you would hit on the best possible model. Every time you made a new guess, it would have the same chance of being the best one, which is hardly the most efficient way of finding the ideal strategy. Ideally, you would make each guess better than the last. This means finding an approach that includes a form of memory.
DURING THE EARLY TWENTIETH century, Poincaré and Borel weren’t the only researchers curious about card shuffling. Andrei Markov was a Russian mathematician with a reputation for immense talent and immense temper. When he was young, he’d even picked up the nickname “Andrei Neistovy”: Andrei the angry.
In 1907, Markov published a paper about random events that incorporated memory. One example was card shuffling. Just as Thorp would notice decades later, the order of a deck after a shuffle depends on its previous arrangement. Moreover, this memory is short-lived. To predict the effect of the next shuffle, you only need to know the current order; having additional information on the cards’ arrangement several shuffles ago is irrelevant. Thanks to Markov’s work, this one-step memory has become known as the “Markov property.” If the random event is repeated several times, it’s a “Markov chain.” From card shuffling to Chutes and Ladders, Markov chains are common in games of chance. They can also help when searching for hidden information.
Remember how it takes at least six dovetail shuffles to properly mix up a deck of cards? One of the mathematicians behind that result was a Stanford professor named Persi Diaconis. A few years after Diaconis published his card shuffling paper, a local prison psychologist turned up at Stanford with another mathematical riddle. The psychologist had brought a bundle of coded messages, confiscated from prisoners. Each one was a jumble of symbols made from circles, dots, and lines.
Diaconis decided to give the code to one of his students, Marc Coram, as a challenge. Coram suspected that the messages used a substitution cipher, with each symbol representing a different letter. The difficulty was working out which letter went where. One option was to tackle the problem through trial and error. Coram could have used a computer to shuffle the letters again and again and then examined the resulting text until he hit upon a message that made sense. This is the Monte Carlo method. He would have deciphered the messages eventually, but it could have taken an absurdly long time to get there.
Rather than starting with a new random guess each time, Coram instead chose to use the Markov property of shuffling to gradually improve his guesses. First, he needed a way to measure how realistic a particular guess was. He downloaded a copy of War and Peace to find out how often different pairs of letters appeared together. This let him work out how common each particular pairing should be in a given piece of text.
During each round of guessing, Coram randomly switched a couple of the letters in the cipher and checked whether his guess had improved. If a message contained more realistic letter pairings than the previous guess, Coram stuck with it for the next go. If the message wasn’t as realistic, he would usually switch back. But occasionally he stuck with a less plausible cipher. It’s a bit like solving a Rubik’s Cube. Sometimes the quickest route to the solution involves a step that at first glance takes you in the wrong direction. And, like a Rubik’s Cube, it might be impossible to find the perfect arrangement by only taking steps that improve things.
The idea of combining the power of the Monte Carlo method with Markov’s memory property originated at Los Alamos. When Nick Metropolis first joined the team in 1943, he’d worked on the problem that had also puzzled Poincaré and Borel: how to understand the interactions between individual molecules. It meant solving the equations that described how particles collided, a frustrating task given the crude calculators around at the time.
After years of battling with the problem, Metropolis and his colleagues realized that if they linked the brute force of the Monte Carlo method with a Markov chain, they would be able to infer t
he properties of substances made of interacting particles. By making smarter guesses, it would be possible to gradually uncover values that couldn’t be observed directly. The technique, which became known as “Markov chain Monte Carlo,” is the same one Coram would later use to decipher the prison messages.
It eventually took Coram a few thousand rounds of computer-assisted guessing to crack the prison code. This was vastly quicker than a pure brute force method would have been. It turned out that one of the prisoners’ messages described the unusual origins of a fight: “Boxer was making loud and loud voices so I tell him por favour can you kick back homie cause I’m playing chess.”
To break the prison code, Coram had to take a set of unobserved values (the letters that corresponded to each symbol) and estimate them using letter pairings, which he could observe. In horse racing, betting teams face a similar problem. They don’t know how much uncertainty surrounds each horse, or how much each factor should contribute to predictions. But—for a particular level of uncertainty and combination of factors—they can measure how well the resulting predictions match actual race outcomes. The method is classic Ulam. Rather than trying to write down and solve a set of near-impenetrable equations, they let the computer do the work instead.
In recent years, Markov chain Monte Carlo has helped syndicates come up with better race forecasts and predict lucrative exotic results like the triple trio. Yet successful gamblers don’t just need to find an edge. They also need to know how to exploit it.
IF YOU WERE BETTING $1.00 on a coin toss coming up tails, a fair payout would be $1.00. Were someone to offer you $2.00 for a bet on tails, that person would be handing you an advantage. You could expect to win $2.00 half the time and suffer a $1.00 loss the other half, which would translate into an expected profit of $0.50.
How much would you bet if someone let you scale up such a biased wager? All of your money? Half of it? Bet too much, and you risk wiping out your savings on an event that still has only a 50 percent chance of success; bet too little, and you won’t be fully exploiting your advantage.
The Perfect Bet Page 7