The Quants: How a New Breed of Math Whizzes Conquered Wall Street and Nearly Destroyed It

by Scott Patterson

  As the Germans closed in, the Mandelbrot family fled to the small hill town of Tulle in southwest France, where they had friends. Benoit enrolled in the local school, where there was little competition. The freedom from the fierce head-to-head pressure of Paris nurtured his creative side. He soon developed the unique ability to picture complex geometric images in his mind and make intuitive leaps about how to solve difficult equations.

  Mandelbrot’s father, a clothing wholesaler, had no job, and the family was destitute. He knew a shopkeeper who had a bundle of coats from before the war with a strange Scottish design. The coats were so hideous that the shopkeeper had trouble giving them away. The senior Mandelbrot took one for his son, who welcomed it.

  One day a group of French partisans blew up a nearby German outpost. A witness noticed that one of the attackers wore a strange-looking jacket with a Scottish design—the same jacket young Mandelbrot wore around the town. When a villager denounced him, he went into hiding, joined by his brother. During the next year, Mandelbrot, innocent of the attack, managed to avoid the German patrols. By the time Allied troops liberated Paris in 1944, he was twenty years old.

  Those nomadic years spent in the countryside of France were crucial in the development of Mandelbrot’s approach to mathematics. The absence of strict guidelines and competition from peers created an environment in which his mind could freely explore the outer limits of mathematical territories most students his age could never dream of.

  He took the entrance examination for Paris’s elite institutions of higher education, the École Normale Supérieure and the École Polytechnique. With no time to prepare, he took it cold. The mathematical section of the test was a complex riddle involving algebra and geometry in which the result (after a great deal of calculation) comes out to zero. Mandelbrot landed the highest score in the country, earning him a ticket to either school. He completed his Ph.D. in 1952.

  After graduating, Mandelbrot entered a period of professional limbo, working for a time with the French psychologist Jean Piaget before spending a year at Princeton’s Institute for Advanced Study in 1953.

  In 1958, he took a job at IBM’s Thomas J. Watson Research Center, the company’s primary laboratory north of Manhattan. By then, his work on issues such as income distributions in various societies had captured the attention of economists outside the cloistered IBM research lab, and in 1961 he went to give a talk at Harvard. When he arrived on campus, he made a beeline to the office of his host for the event, the economics professor Hendrik Houthakker. Soon after entering, he was stunned by a strange diagram on the professor’s blackboard, a convex V that opened out to the right. Mandelbrot sat down. The image on the blackboard loomed over Houthakker’s shoulder. Mandelbrot couldn’t keep his eyes off it.

  “I’m sorry,” he said after a few minutes’ chitchat. “I keep looking at your blackboard because this is a strange situation. You have on your blackboard a diagram from my lecture.”

  Houthakker turned and gazed at the diagram. “What do you mean?” he said. “I have no idea what you’re going to talk about.”

  The diagram came from a student’s research project on the behavior of cotton prices, an obsession of Houthakker’s. The student was trying to discern how the patterns in cotton prices fit into the standard Brownian motion models that dominated financial theory. But to his great frustration, nothing worked. The data didn’t fit the theory or the bell curve. Prices flitted about too erratically. The stunning coincidence for Mandelbrot was that the diagram of cotton prices on Houthakker’s chalkboard exactly matched the diagram of income distributions Mandelbrot had prepared for his talk.

  The bizarre leaps and plunges in cotton prices had proved too wild for Houthakker. Either the data were bad—unlikely, as there were a lot of data, going back more than a century from records kept by the New York Cotton Exchange—or the models were faulty. Either way, he was on the verge of giving up.

  “I’ve had enough,” he told Mandelbrot. “I’ve done everything I could to make sense of these cotton prices. I try to measure the volatility. It changes all the time. Everything changes. Nothing is constant. It’s a mess of the worst kind.”

  Mandelbrot saw an opportunity. There might be a hidden relationship between his own analysis of income distributions—which also displayed wild, disparate leaps that didn’t fall within the normal bell curve—and these unruly cotton prices that had driven Houthakker to his wits’ end. Houthakker happily handed over a cardboard box full of computer punch cards containing data on cotton prices.

  “Good luck if you can make any sense of these.”

  Upon returning to IBM’s research center in Yorktown Heights, Mandelbrot began running the data through IBM’s supercomputers. He gathered prices from dust-ridden books at the National Bureau of Economic Research in Manhattan and from the U.S. Agriculture Department in Washington. He looked into wheat prices, railroad stocks, and interest rates. Everywhere he looked he saw the same thing: huge leaps where they didn’t belong—on the outer edges of the bell curve.

  After combing through the data, Mandelbrot wrote a paper detailing his findings, “The Variation of Certain Speculative Prices.” Published as an internal research report at IBM, it was a direct attack on the normal distributions used to model the market. While praising Louis Bachelier, a personal hero of Mandelbrot’s, the mathematician asserted that “the empirical distributions of price changes are usually too ‘peaked’ relative to samples” from standard distributions.

  The reason: “Large price changes are much more frequent than predicted.”

  Mandelbrot proposed an alternative method to measure the erratic behavior of prices, one that borrows a mathematical technique devised by the French mathematician Paul Lévy, whom he’d studied under in Paris. Lévy investigated distributions in which a single sample radically changes the curve. The average of the heights of 1,000 people won’t change very much as a result of the height of the 1,001st person. But a so-called Lévy distribution can be thrown off by a single wild shift in the sample. Mandelbrot uses the example of a blindfolded archer: 1,000 shots may fall close to the target, but the 1,001st shot, by happenstance, may fall very wide of the mark, radically changing the overall distribution. It was an entirely different way of looking at statistical patterns—all previous results could be overturned by one single dramatic shift in the trend, such as a 23 percent drop in the stock market in a single day. Lévy’s formulas gave Mandelbrot the mathematical key to analyzing the wild moves in cotton prices that had befuddled Houthakker.

  When plotted on a chart, these wild, unexpected moves looked nothing like the standard bell curve. Instead, the curve bubbled out on both ends, the “tails” of the distribution. The bubbles came to be known as “fat tails.”

  Word of Mandelbrot’s paper spread through the academic community. In late 1963, he got a call from Paul Cootner, an MIT finance professor. Cootner was putting together a book of published material on recent mathematical insights into the workings of the market, including a translation of Bachelier’s thesis on Brownian motion. He wanted to include Mandelbrot’s paper. He called the book The Random Character of Stock Market Prices. It was the same book Ed Thorp read a year later when he was trying to figure out a formula to price warrants.

  In the book, Cootner attacked Mandelbrot’s submission in a vicious five-page critique. Mandelbrot “promises us not utopia but blood, sweat, toil, and tears.” The wild gyrating mess of Lévy’s formulas, the sudden leaps in prices, simply wouldn’t do. The result would be chaos. While several economists briefly glommed on to Mandelbrot’s analysis, it soon fell out of favor. Some said the approach was too simplistic. Others simply found the method too inconvenient, incapable of predicting prices, as if one were trying to forecast the direction of a Mexican jumping bean. Critics said that while it may work for brief time periods when price action can be erratic, over longer time periods, prices appear to move in a more orderly Brownian fashion. An eyeball test of long-term trends in the stoc
k market shows that prices of an entire market do tend to move in more regular, less erratic patterns.

  Mandelbrot agreed that over long periods, equilibrium tends to rule the day. But that misses the point. Prices can gyrate wildly over short periods of time—wildly enough to cause massive, potentially crippling losses to investors who’ve made large, leveraged wagers.

  As Nassim Nicholas Taleb, a critic of quant models, later argued in several books, investors who believe the market moves according to a random walk are “fooled by randomness” (the title of one of his books). Taleb famously dubbed the wild unexpected swings in markets, and in life itself, “black swans,” evoking the belief long held in the West that all swans are white, a notion exploded when sailors discovered black swans in Australia. Taleb argued that there are far more black swans in the world than many people believe, and that models based on historical trends and expectations of a random walk are bound to lead their users to destruction.

  Mandelbrot’s theories were shelved by the financial engineers who didn’t want to deal with the messy, chaotic world they evoked. But they always loomed like a bad memory in the back of their minds, and were from time to time thrust to the forefront during wild periods of volatility such as Black Monday, only to be forgotten again when the markets eventually calmed down, as they always seemed to do.

  Inevitably, though, the deadly volatility returns. About a decade after Black Monday, the math geniuses behind a massive quant hedge fund known as Long-Term Capital Management came face-to-face with Mandelbrot’s wild markets. In a matter of weeks in the summer of 1998, LTCM lost billions, threatening to destabilize global markets and prompting a massive bailout organized by Fed chairman Alan Greenspan. LTCM’s trades, based on sophisticated computer models and risk management strategies, employed unfathomable amounts of leverage. When the market behaved in ways those models never could have predicted, the layers of leverage caused the fund’s capital to evaporate.

  The traders behind LTCM, whose partners included option-formula creators Myron Scholes and Robert Merton, have often said that if they’d been able to hold on to their positions long enough, they’d have made money. It’s a nice theory. The reality is far more simple. LTCM went all in and lost.

  Black Monday left an indelible stamp on the very fabric of the market’s structure. Soon after the crash, options traders started to notice a strange pattern on charts of stock-option prices. Prices for deep out-of-the-money puts—long-shot bets on huge price declines—were unusually high, compared with prices for puts closer to the current price of the stock. Graphs of these prices displayed a curvy kink around such options that, according to the prevailing theory, shouldn’t exist. Traders soon came up with a name for this phenomenon: the “volatility smile.” It was the grim memory of Black Monday grinning sinisterly from within the very prices that underpinned the market.

  The volatility smile disobeyed the orderly world of “no arbitrage” laid out by Black-Scholes and modern portfolio theory, since it implied that traders could make a lot of money by selling these out-of-the-money puts. If the puts were too expensive for the risk they carried (according to the formula), the smart move would be to sell them hand over fist. Eventually that would drive the price down to where it should be. But, oddly, traders weren’t doing that. They were presumably frightened that another crash like Black Monday could wipe them out. They never got over the fear. The volatility smile persists to this day.

  The volatility smile perplexed Wall Street’s quants. For one thing, it made a hash of their carefully calibrated hedging strategies. It also raised questions about the underlying theory itself.

  “I realized that the existence of the smile was completely at odds with Black and Scholes’s 20-year-old foundation of options theory,” wrote Emanuel Derman, a longtime financial engineer who worked alongside Fischer Black at Goldman Sachs, in his book My Life as a Quant. “And, if the Black-Scholes formula was wrong, then so was the predicted sensitivity of an option’s price to movements in its underlying index. … The smile, therefore, poked a small hole deep into the dike of theory that sheltered options trading.”

  Black Monday did more than that. It poked a hole not only in the Black-Scholes formula but in the foundations underlying the quantitative revolution itself. Stocks didn’t move in the tiny incremental ticks predicted by Brownian motion and the random walk theory. They leapt around like Mexican jumping beans. Investors weren’t rational, as quant theory assumed they were; they panicked like rats on a sinking ship.

  Worse, the engine behind the crash, portfolio insurance, was the spawn of the quants, a product designed to protect investors from big losses. Instead, it created the very losses it was meant to avoid.

  Not everyone suffered catastrophic losses on Black Monday. Princeton/Newport Partners, due to Thorp’s fancy footwork, lost only a few million dollars that day. After the crash, Thorp’s models, scanning the marketplace like heat-seeking missiles, sought out numerous good deals. The fund closed the month flat. For the year, the fund earned a 27 percent return, compared with a 5 percent gain by the S&P 500.

  Thorp had managed to survive the most devastating drawdown in the history of the stock market. Everything was looking up. Then, out of the blue, disaster struck Princeton/Newport Partners. It was Ed Thorp’s black swan.

  In mid-December 1987, an army of vans pulled up in front of a nondescript office complex in the heart of sleepy Princeton. A squad of fifty armed federal marshals clad in bulletproof vests burst from the vans and rushed into the office of Princeton/Newport Partners, which was perched in a small space over a Häagen-Dazs shop.

  They were searching for documents related to the fund’s dealings with Michael Milken’s junk bond empire at Drexel Burnham Lambert. The man in charge of the case was Rudolph Giuliani, the U.S. attorney for the Southern District of New York. He was trying to build more evidence for the government’s case against Drexel and was hoping employees of the hedge fund, threatened with stiff fines and possible prison terms, would turn against Milken.

  It didn’t work. In August 1989, a Manhattan jury convicted five Princeton/Newport executives—including Regan—of sixty-three felony counts related to illegal stock trading plots. Thorp, more than two thousand miles away at the Newport Beach office and oblivious to the alleged dark dealings in the fund’s Princeton headquarters, was never charged. But Regan and the other convicted partners at Princeton/Newport refused to testify against Milken or acknowledge wrongdoing. Instead, they fought the government’s charges—and won. In June 1991, a federal appeals court tossed out the racketeering convictions in the government’s fraud case. Early the following year, prosecutors dropped the case. Not a single employee of Princeton/Newport spent a day in prison.

  The biggest casualty of the government’s assault was Princeton/Newport. It became impossible for Thorp to keep the ship steady amid all the controversy, and his associates in Princeton were obviously distracted dealing with the charges against them. Worried investors pulled out of the fund.

  Thorp decided to simplify his life. He took a brief break from managing money for others, though he continued to invest his own sizable funds in the market.

  He also worked as a consultant for pension funds and endowments. In 1991, a company asked Thorp to look over its investment portfolio. As he combed through the various holdings, he noticed one particular investment vehicle that had produced stunning returns throughout the 1980s. Every single year, it put up returns of 20 percent or more, far outpacing anything Thorp had ever seen—even Princeton/Newport. Intrigued, and a bit dubious, he delved further into the fund’s strategies, requesting documents that listed its trading activities. The fund, based in New York’s famed Lipstick Building on Third Avenue, supposedly traded stock options on a rapid-fire basis, benefiting from a secret formula that allowed it to buy low and sell high. The trading record the fund sent Thorp listed the trades—how many options it bought, which companies, how much money it made or lost on the trades.

 
It took Thorp about a day to realize the fund was a fraud. The number of options it reported having bought and sold far outpaced the total number traded on public exchanges. For instance, on April 16, 1991, the firm reported that it had purchased 123 call options on Procter & Gamble stock. But only 20 P&G options in total had changed hands that day (this was well before the explosion in options trading that occurred over the following decade). Similar discrepancies appeared for trades on IBM, Disney, and Merck options, among others, Thorp’s research revealed. He told the firm that had made the investment to pull its money out of the fund, which was called Bernard L. Madoff Investment Securities.

  In late 2008, the fund, run by New York financier Bernard Madoff, was revealed as the greatest Ponzi scheme of all time, a massive fraud that had bilked investors out of tens of billions. Regulators had been repeatedly warned about the fund, but they never could determine whether its trading strategies were legitimate.

  While Thorp was taking a break from the investing game, the stage for the amazing rise of the quants had been set. Peter Muller, working at a quant factory in California, was itching to branch out and start trading serious money. Cliff Asness was entering an elite finance program at the University of Chicago. Boaz Weinstein was still in high school but already had his eyes trained on Wall Street’s action-packed trading floors.

  As Thorp wound down Princeton/Newport Partners, he handed off his hedge fund baton to a twenty-two-year-old prodigy who would go on to become one of the most powerful hedge fund managers in the world—and who would play a central role in the market meltdown that began in August 2007.

  GRIFFIN