The two great problems of meteorology remain data collection and interpretation. Bjerknes had opened up the prospect of deterministic forecasts, if only there was enough information to feed into his equations. Accurate prediction seemed to require Laplace’s omniscient observer, who could tell you all the future, but only if you could tell him everything about the present. Was there at least a sample on which to try these powerful equations—one moment sufficiently documented to justify wrestling with their nonlinear terms?
There was: May 20, 1910, turned out to be a vital day for the history of weather forecasting. Nothing important happened in the weather itself, but a coordinated set of balloon flights across Europe had brought back a dense, consistent data set describing winds, temperature, and pressure at equal altitudes and times. If it were ever going to be possible to treat the weather purely as a math problem, here was one exercise, at least, that included the answer from the back of the book.
The man who took up the challenge was Lewis Fry Richardson, another of history’s stubborn and persistent Quakers, who, ardently pacifist, reserved his aggressive instincts for attacking columns of figures. His work on underground flow in peat bogs had made him familiar with the hydrodynamical equations that Bjerknes used; while his prior mathematical training convinced him that it should be possible, if not to solve them, at least to approximate their solution through the elegantly named “method of finite differences.”
When the war came, Richardson registered as a conscientious objector and went to France to serve with the Friends’ Ambulance Unit. He took with him the great May 20 problem: to project, in purely numerical terms, six hours’ progress of that calm, unexceptional spring day when all Europe had still been at peace. It took him the best part of six weeks to work out the distribution, his office a heap of hay in a cold barn. Richardson sent his manuscript to the rear, out of danger—where it disappeared, only to be found a month later under a pile of coal.
This first attempt at numerical weather forecasting was not a complete success: Richardson’s calculations retrospectively predicted that May 20, 1910, having begun so calmly, would go on to see violent weather patterns surging out of the East on winds faster than the speed of sound—a normal day on Jupiter, but not in Europe. Nevertheless, he realized that the error lay in the application rather than the principle. The calculations, even using the method of finite differences, were simply too complex and finicky for one person. They would need to be shared out—and here he had his vision of The Weather Factory:
Imagine a large hall like a theatre, except that the circles and galleries go right round through the space usually occupied by the stage. The walls of this chamber are painted to form a map of the globe. The ceiling represents the north polar regions, England is in the gallery, the tropics in the upper circle, Australia on the dress circle and the antarctic in the pit. A myriad computers are at work upon the weather of the part of the map where each sits. . . . The man in charge of the whole theatre . . . turns a beam of rosy light upon any region that is running ahead of the rest, and a beam of blue light upon those who are behindhand.
The “computers,” of course, would be people, like Karl Pearson’s young women—each one assigned a separate, specialist’s portion of one calculation of one equation, each group of 32 working to generate a new data point in the three hours before the weather caught up with them. Assuming that the world were divided into a grid of squares 200 kilometers on a side, you would need around 64,000 of these computers, all rattling away on their adding machines. Richardson knew this was fantasy; but, significantly, it was accurate fantasy. By scaling up from his own conscientious attempts to forecast weather, he was able to specify, within an order of magnitude, the mass of data and power of calculation that would be necessary to hoist weather prediction out of the slough of proverb and analogy.
Conflict concentrates minds and resources. At the beginning of the First World War, armies had not considered weather as anything more than a tactical nuisance; forecasting was of little importance since, as one British general allegedly put it, “Troops don’t march into battle with umbrellas.” The murderous vagaries of poison gas, driven back on fluctuating winds, soon changed their minds. When the Second World War came around, the necessity for air cover and long-range planning of joint operations revived a demand for accurate forecasts—but these were still difficult to provide. Warren Weaver, head of America’s war mathematicians, reckoned that it was actually inaccurate forecasting that helped ensure the success of the D-Day landings:
the weather forecast for that event, made by the American forces, was so inconceivably bad that the German meteorological experts, who were substantially better, simply couldn’t believe that we would be so stupid as to make so bad a forecast, and could not believe that we would act upon it, and therefore could not believe that the invasion would occur at the time when it actually did.
But at that very moment, in the scientific shantytown built for the Manhattan Project, America’s full financial and intellectual power was being applied to the problem of non-linear differential equations—because the same complex dynamics of interdependent variables that create weather are also at work in the searing heart of an atomic bomb. Newton’s laws plus thermodynamics apply as surely to the pressure front of the implosion that initiates fission as to the pressure front that brings rain in the night, only the values are much greater and the timescale much faster. It was to help describe these instantaneous, ferocious tempests that John von Neumann, godfather of modern computing, was invited to Los Alamos.
Until his arrival, work on the equations had proceeded much as Lewis Richardson had imagined: human computers (usually the wives of the physicists) sat at desk calculators, repetitively churning out subsections of finite-difference solutions. Von Neumann, though, had already watched the first electronic computers in action. Here, at last, was a calculating power to rival the great spherical hall.
Von Neumann, described as having a mind like a racing car among tricycles, immediately saw a concatenation of possibilities: the new computers, financed by the war effort, would not only support postwar military planning but also take on the challenges of civil life, including predicting—nay, controlling—the weather. All went well, at least with the prediction side: in 1945, it took six days to produce a reasonably accurate 24-hour forecast of atmospheric pressure over North America—still, therefore, predicting last week’s weather. In 1950, computers had achieved Richardson’s goal of keeping pace with real time. By 1952, 24 hours of prediction took just five minutes of calculation. Soon, von Neumann and his colleagues believed, it would be possible to take advantage of the non-linearity (and thus the sensitivity) of weather to influence it in favorable ways—to seed clouds and put rain where it was needed, to steer storms, to smooth out the haphazardness of the skies. In that fast-receding decade of great optimism and great fear, of dream homes with fallout shelters, it seemed the most basic uncertainty of this world was about to disappear forever.
This is a story often told because it is so good: like all the best scientific anecdotes, it contains both accident and unwillingness to accept the merely accidental. In 1961, Edward Lorenz had a computer running weather simulations in his office at MIT. It was a Royal McBee: as big as a desk and expensive as a house, but with roughly the processing power of a modern pocket calculator, so Lorenz vastly simplified the equations for the weather of its virtual world. There was no friction, there was no moisture, there were no seasons; yet he could see in the pattern of winds and pressure that circled his private globe things that looked familiar from his days as a weather observer in the Army Air Corps: fronts progressing from west to east, circular storms.
One day that winter, Lorenz decided to extend a simulation he had run previously. To give the machine a decent run-up, he put in as a starting point the values it had generated halfway through the earlier sequence. Lorenz headed down the corridor for a cup of coffee, and when he returned, he found an oddity: the weather newly
generated for the repeated half of the run differed entirely from the first version. That is, the same input, entered into the same equations, was generating a widely divergent output.
What was happening inside the Royal McBee? Lorenz realized that although his machine printed out numbers to only three decimal places, it calculated using six. So the printed-out numbers he had entered to restart the run must have differed from the actual halfway figures, although only by tens of thousandths. These tiny differences—significantly smaller than the resolution of most real weather observation—were enough to produce entirely different results within only a few simulated months. Lorenz says that he realized at that moment that long-range weather forecasting was impossible.
The formal name for this is sensitivity to initial conditions. Billiard balls have always been used as the familiar model for a determinist, Newtonian physical system; but in America, we have bumper pool, which aims to allow the full variety of shots, finessed or fluffed, by putting circular elastic obstacles in the middle. The round bumpers amplify the difference between one angle of incidence and another: a degree’s variation in the path of the ball off the cue becomes, say, five degrees after the first bounce, then twenty five on the next, which probably puts your ball in a line to hit something entirely unexpected. You lose through what looks like bad luck: sensitivity to initial conditions.
Lorenz’s discovery means that we now know that deterministic systems—ones with clear rules that apply always and everywhere—come in two radically different forms. The first is like flipping a coin: you may not know exactly what will happen in any single instance, but over many instances variation will cancel out and a clear pattern will emerge. Repetition allows prediction. The other is like bumper pool: small variations aggregate and amplify—repetition forbids prediction. These latter systems are what we now call chaotic.
The mathematics of chaotic systems produces the same effect at every scale. Tell me how precise you want to be, and I can introduce my little germ of instability one decimal place farther along; it may take a few more repetitions before the whole system’s state becomes unpredictable, but the inevitability of chaos remains. The conventional image has the flap of a butterfly’s wings in Brazil causing a storm in China, but even this is a needlessly gross impetus. The physicist David Ruelle, a major figure in chaos theory, gives a convincing demonstration that suspending the gravitational effect on our atmosphere of one electron at the limit of the observable universe would take no more than two weeks to make a difference in Earth’s weather equivalent to having rain rather than sun during a romantic picnic.
Lorenz provided a picture that remains the icon of chaos: a basic example of what are now called strange attractors. It shows, in three dimensions, the history of a simple system, generated by three basic non-linear equations. It has two general states, often described as the two wings of a butterfly. You could postulate, if you were a meteorologist, that the left state describes the general weather when the high-altitude jet stream is toward the north, bringing cold, clear air down from the Arctic; while the right describes the conditions when the jet stream has shifted south, bringing wetter, unsettled weather. The state of the system at any given moment is represented by a point on this trajectory: it may stay generally under the sway of the left-hand state for a few turns, or it may suddenly switch, during this turn, to the right-hand state. Similarly, any two points—as close together as you like—taken as the starting point for this path may stay close for a turn or two but will then suddenly diverge; just as Darwin and Fitzroy, confined to the same trajectory for their five years’ voyage, ended up, the one changing our understanding of the world, the other caught in its baffling detail.
Chaos theory did not just spring full-armed from the brow of Ed Lorenz. Back in the fourteenth century, the wonderful French bishop Nicole d’Oresme, who came close to realizing so many later scientific ideas, had a glimmering of it in his rejection of astrology: he argued that one could make no reasonable prediction of human fortune based on the positions of the stars and planets, not just because of the lack of necessary causation, but because the planetary system itself was structurally unstable—no state was exactly the same as a previous one, nor would it be exactly the same again. The idea resurfaced in the work of an equally great Frenchman, Poincaré, who studied the stability of the solar system and found similar unpredictability in the movements of just three bodies interacting under the laws of gravity. What made Lorenz’s discovery so important was that he had shown chaos to be at work all around us; he defined a limit to our intellectual and technological power—not deep within the atom or at the fringes of the observable universe, but in the view out every window.
Where does this leave weather forecasting? In the invidious position of someone whose undoubted achievements are overshadowed by his limitations: the home-run champion of the minor leagues. Short-term forecasting has improved enormously. In the UK, 24-hour forecasts are accurate 87 percent of the time. In the United States, the reliability of the 36-hour forecast at an altitude of 5 kilometers has improved to 90 percent—good news for pilots, if not for picnickers. A weatherman’s statement of a 60 percent probability of rain overnight is both more carefully calculated and more likely to be true than a doctor’s statement of a 60 percent probability of surviving a particular operation. Yet whom do people trust more?
There are, however, two probabilistic reasons to mistrust weather forecasters. The first has to do with the relative likelihood versus the importance of different kinds of weather: it’s relatively easy to forecast accurately another day of clear weather in a stable high-pressure system, but that’s not a forecast most people notice or remember. Hailstorms and tornadoes are very memorable, but exceedingly difficult to predict for any one place or time: they occupy little space and their genesis is exquisitely sensitive to initial conditions. Even with a dense network of weather data, a sudden squall can thread its way through the observations like a cat through a closing door. This is what happened with the great UK storm of 1987, where the television forecaster squandered a lifetime’s credibility by pooh-poohing a viewer’s worry that a hurricane was on its way, only a few hours before the trees began to crash down in Hyde Park.
The second disadvantage that probability loads onto the forecaster is what’s called the base-rate effect—and here, since prediction involves expectation, we come back to Bayes’ theorem. As you’ll remember, the theorem describes how our previous assumptions about the probability of an event are modified by evidence with a given intrinsic probability (or, in the case of weather forecasting, credibility).
This means our assumptions about forecasting a given event are based on the intrinsic accuracy of the forecast times the intrinsic likelihood of the event itself. So when an event is likely (rain in Bergen), an accurate forecast has a favorable effect on our assumptions. When the predicted event is unlikely, though, it drastically reduces our reasons to believe in any forecast, however accurate. In one celebrated case, the Finley Affair of 1884, a meteorologist claimed that his forecasts of whether a given place would see a tornado or not had an accuracy of 96.6 percent. This sounded very impressive until G. K. Gilbert pointed out that simply putting up a board with “No Tornado” painted on it would have a predictive accuracy of 98.2 percent. As with medical testing, accuracy means less in the context of rarity.
Richardson’s Weather Factory exists, in a sense, as the European Centre for Medium-Range Weather Forecasts, a cooperation between 25 countries anxious to understand the effects of weather. Its meteorologists and supercomputers create deterministic forecasts for the next 3 to 6 days; but they also squarely take on chaos by issuing probabilistic 10-day forecasts for Europe. Here, the butterfly that will or will not create the storm in Spain hovers somewhere over the Northern Pacific. How do the forecasters decide whether it has flapped or not?
The answer is, they don’t. Instead, they determine, based on the current weather, where the areas of greatest sensitivity are, and then vary
parameters for those areas randomly (but within reasonable limits) in the computer, generating an arbitrary 51 alternative starting points on which to run the simulation. What comes out at the end, therefore, is not a single forecast but a probability distribution: look at any point on the map and you will have a curve of 51 possible values for wind speed, temperature, or pressure. If those values cluster closely around a mean in a statistically normal fashion, you can be sure that whatever the butterfly may be planning for this week, it will have little effect on you. If the values scatter randomly, you can be sure at least of uncertainty. It’s called ensemble prediction; it makes predictability a variable, just like temperature or rainfall.
Of course, this isn’t the way most of us are used to thinking about weather. “A mate rang me up on Monday—he was going to have a garden party on Saturday and he wanted to know if it was going to rain.” Tim Palmer, division head at the European Centre, is young, brisk, and fluent. “I said I could give him a probability. ‘I don’t want that—just tell me if it’s going to rain or not.’ I said: ‘Look. Is the Queen coming to your party?’ ‘What’s that got to do with it?’ ‘What’s your risk if it rains and you don’t have a tent? What probability of rain can you tolerate?’ He thought about it and said: ‘It’s just friends from work; I can tolerate a sixty percent probability.’ In fact, the ensemble forecast gave a probability of around thirty-two percent. He didn’t book the tent and it didn’t rain. Thank God.”
The weather simulation on which the ensemble forecasts are run is a long way from the frictionless, seasonless world of Lorenz’s early program. It includes the effects of ocean temperature, wave friction, and mountain ranges. “If you want to study weather nowadays, you need to have very broad knowledge: radiation, basic quantum physics, chemistry, marine biology, fluid mechanics—they’re all involved in how the atmosphere works. The problem when you combine them into one model, though—with its millions and millions and millions of lines of code—is that no one person understands the whole thing anymore.”
Chances Are Page 25