Similarly, when at the bottom, the tendency is often the other way. 'It's a low, low, low rate world' said the cover of Business Week about the cost of borrowing in March 2007, shortly before rates rose sharply, though not quite as comically as the 1979 cover predicting 'The Death of Equities', when the Dow-Jones market index was at 800. In January 2008 it stood at about 12,500. Reporting on this tendency for numbers to go into reverse, The Economist magazine confessed its own prediction in the late 1990s that oil would drop to $5 per barrel. In 2008 it topped $120.
Examples like these tempt us to assume that whatever a business magazine recommends constitutes a sure guide to doing the opposite, being based, as these stories often are, on the most recent up or down, rather than on that more reliable tendency to fluctuation. Alas, there are no sure guides in business, but one study in 2007, 'Are Cover Stories Effective Contrarian Indicators?' by academics at the University of Richmond published in the Financial Analysts Journal, looked at headlines from featured stories in Business Week, Fortune and Forbes magazines from a twenty-year period. It concluded that 'positive stories generally indicate the end of superior performance and negative news generally indicates the end of poor performance'.
Exposing the cover jinx makes an entertaining but exaggerated point. Not everything that goes up and down is due to chance. In truth, there may be underlying reasons that have nothing to do with chance variation (or magic). But in asking: 'why did it happen?' the chance that causes stuff to go up and down is, as with clusters, the potential explanation most often ignored.
It is not only entertaining. Imagine a forty-year diet of whipped cream and two packets of cigarettes daily. Your arteries would quite likely be as well lined as your furry slippers. A heart attack looms.
Heart attacks are often like that: a long time coming, with causes that tend to be life-long, the culprits partly genetic, partly dietary, frequently including smoking, stress and lack of exercise – all well-known suspects – and damage that is cumulative. It peaks the day of a good lunch or a tight game on TV, when the quantity of oxygen reaching the heart muscles finally becomes insufficient and …
Except, that is, if you live in the small town of Helena, Montana. Here, according to a now famous study, a large proportion of heart attacks have been explained not by people's own habits alone, but partly by those of others. It is breathing cigarette smoke in the atmosphere – passive smoking – that is often to blame. Or so it is said.
The study in Helena was astonishing. It found a fall in heart attacks at the local hospital of 40 per cent during a six-month ban on smoking in public places, compared with the same six months previously. When the ban ended, the number of heart attacks went up again. The research was published in the British Medical Journal.
We had better declare that neither of us is a pro- nor antismoking lobbyist or campaigner. Neither of us is a smoker, nor wants to breathe other people's smoke, but that preference can't disguise a problem with the Helena study: it overlooks the fact that numbers also go up and down by chance.
Can chance really explain a 40 per cent drop that bounces straight back up? As with clusters, people fidget for meaning whenever numbers rise and fall. What made them rise; what made them fall? It is the right question to ask, but we might be disappointed by the answer. Numbers can go up and down for no more reason than occurs when chance varies the height of the waves rolling into shore. Sometimes waves crash, sometimes they ripple, now larger, now smaller. This succession of irregular peaks and troughs, swelling and sinking, fussy and dramatic, we'll call chance. Like chance it has its causes, but they are so intricately tangled we can almost never unravel them. Imagine those waves like numbers that are sometimes high and sometimes low. It is the easiest mistake in the world to observe a big wave, or a high number, and assume that things are on the up. But we all know that a big wave can arrive on a falling tide. This is no great insight. Everyone knows it from life. The suggestion here is simply that what is known in life is used in numbers. The tendency among some to feel that numbers are alien to experience, incomprehensible, inaccessible, a language they will never speak, should be put aside. In truth, they speak it already. Numbers often work on principles that we all use with easy familiarity, all the time.
And so we can easily see that a momentarily high number – in accident rates or elsewhere – can, just like a big wave, be part of a falling trend. And yet taking a measurement from a single peak and assuming it marks a rising tide, though a simple error, is one that in politics really happens. It is assumed that the peak in the numbers tells us all we need to know, that it happened for a critical reason, caused by an action we took or a policy we implemented, when really it is just chance that made the numbers momentarily swell. Is this obvious? It is hard to believe that politics sometimes needs lessons from the rest of us in the view from the beach, but it is true. For to know if the explanation is accurate, what matters is not the size of the wave at all, but rather the tide, or maybe even the sea level. The wave catches the eye, but look elsewhere before being sure of capturing a real or sustained change, not a chance one. Otherwise, you will fail to distinguish cause and effect from noise.
But to repeat the acid question, can this explain the Helena study? The chart below shows deaths from heart attacks in Lewis and Clark County, Montana. About 85 per cent of the county population lives in Helena. These are deaths from heart attacks, not heart attack admissions, but represent the best data we can find covering a sufficiently long period, before the smoking ban took effect, in June 2002. The chart also first appeared in the BMJ, submitted by Brad Rodu, professor of pathology at the University of Alabama medical school.
Two points stand out: the downward trend (the tide), and that the numbers go up and down around this trend (the waves). And when, by chance, they have gone down sharply in one year, the next move tends to be up. We see falls from one year to the next of about 30 or 40 per cent in 1980, 1988, 1992, 1994 and 2001.
Figure 4 Age-adjusted mortality rate for acute myocardial infarction among persons aged 35+ years, Lewis and Clark County, Montana, 1979–2001
In St Peter's hospital, Helena, the number of heart attack patients admitted during the studied period was tiny: an average of four a month during the ban, six or seven a month beforehand. It is worth adding that most of the fall occurred in the first half of the ban, and that the numbers rose again in the second half. The study was right to be cautious in concluding that laws to enforce smoke-free workplaces and public places 'may be associated with an effect on morbidity from heart disease' (emphasis ours).
Did the ban on smoking in public places in Helena reduce heart attacks by 40 per cent? We doubt it. This looks more like the frisky dog around the man than the man himself. The statistical intelligence of studies on the effect of such bans around the world has been generally poor. A widely reported study in Scotland, based on a sample of hospitals, asserted that a similar ban there reduced heart attack admissions by 17 per cent, another huge change. That is, until official data for all admissions showed the extent of the fall to be about 8 per cent, and this against a recent downward trend before the ban of 5 or 6 per cent, and a fall seven years earlier of about 11 per cent. The fact that there have been several other initiatives designed to reduce heart attacks in Scotland in the last few years makes it hard to say for sure what, if any, effect the ban had in Scotland. Though we would not be at all surprised if it had some substantially smaller effect than that claimed by researchers, and we'd expect that to become clearer with time.
Time would be a good test of whether the fall was a frisky dog, a chance, short-term up-and-down, or a real change, but the ban in Helena ended after six months. It has been reported that a ban in California – together with steep increases in tobacco taxes and an anti-smoking campaign – coincided with a fall in deaths from heart attack about 5 or 6 per cent greater than elsewhere in the US, but this was the cumulative total over a period of nine years. Compare that with a claim of a 40 per cent fall in h
eart attacks in six months from a ban alone, based on a few handfuls of cases.
The inevitability of variation in numbers like this regularly sinks understanding, as people race to over-interpret what are often chance results A contentious and none-too-obvious example is the roadside speed camera. These are deadly devices, if not to traffic then certainly to conversation. Oppose them and you crave toddlers at play slaughtered; you seek to scream unimpeded through residential areas pushing 90mph. Support them, and it is owing to your control-freak loathing for others' freedom and pig-ignorance of the facts about a flawed system. The antis are caricatured as don't-give-a-damn evaders of civic responsibility, the pros of wanting the motorist fined off the road by a stealth tax to fill the coppers' coffers. According to the Daily Telegraph (10 August 2006), 'a survey found that 16 per cent of people support the illegal destruction of speed cameras by vigilante gangs'.
By Luke Traynor – Liverpool Daily Post
MERSEYSIDE'S speed cameras are picking up motorists driving at up to 134mph, figures revealed yesterday. In one incident, a driver was caught racing at 134mph in a 50mph zone, nearly three times over the limit, on the M62 near the Rocket public house. Over a six-month period 116 motorists were caught going at or above 70mph in a 30mph zone.
Three cheers for speed cameras? Then again …
By Philip Cardy – Sun
Nicked for doing 406mph
Driver Peter O'Flynn was stunned to receive a speeding notice claiming a roadside camera had zapped him – at an astonishing 406mph. The sales manager, who was driving a Peugeot 406 at the time, said: 'I rarely speed and it's safe to say I'll contest this.'
With indignant voices on either side, sane discussion is perilous, but why is there even argument? Judgement ought to be easy based on the numbers: are there more accidents with cameras or without? The answer seems clear: on the whole, accidents decline when cameras are installed, sometimes dramatically.
Department for Transport Press Release, 11 February 2003
Deaths and serious injuries fell by 35% on roads where speed cameras have been in operation, Transport Secretary Alistair Darling announced today. The findings come from an independent report of the two-year pilot scheme where eight areas were allowed to re-invest some of the money from speeding fines into the installation of more cameras and increased camera use.
Transport Secretary Alistair Darling said: 'The report clearly shows speed cameras are working. Speeds are down and so are deaths and injuries … This means that more lives can be saved and more injuries avoided. It is quite clear that speeding is dangerous and causes too much suffering. I hope this reinforces the message that speed cameras are there to stop people speeding and make the roads safer. If you don't speed, you won't get a ticket.'
This 35 per cent cut in deaths and serious injuries equated, the department said, to 280 people, and since then the number of cameras has grown. Case closed? Not quite. First, there are exceptions: at some sites the number of accidents went up after speed cameras arrived. But a minority of perverse results will not win the argument, we need to know what happens on balance: on balance there was still a large fall.
The second reservation risks sounding complicated, but is in principle easy: it is the need to distinguish between waves and tides, or the dog and the man (which for some years the Department for Transport preferred not to do). As with waves, the rate at which things happen in the world goes up and down, though without telling us much about the underlying trends: there are more this time, fewer next, rising one week, falling another, simply by chance. People especially do not behave with perfect regularity. They do not sort themselves into an even flow, do not get up, go to work, eat, go out, catch the bus, crash, either at the same time or at even intervals. The number doing any of these things at any one time goes up and down.
Of course, you knew that already. Accident statistics at any one site will quite likely go up and down from time to time, because that is what they do – a bad crash one month, nothing the next. The freakish result would be if there were exactly the same number of accidents in any one place over every twelve-month period. Luck, or bad luck, often determines the statistics, as the wave rises and falls.
The argument bites with the realisation that, all else being equal, if the numbers have been higher than usual lately, the next move is more likely to be down than up. After a large wave often comes a smaller one, the dog goes one way then the other. Two bad crashes last month on an unexceptional stretch of road and, unless there is some obvious hitch, we would be surprised if things did not settle down.
Applied to road accidents, the principle has this effect: put a speed camera on a site where the figures have just gone up, at the crest of a big wave – and this does tend to be when the cameras arrive, in the belief that these sites have been newly identified as a problem – and the next move in the accident statistics is likely to be down – whether the speed camera is there or not. The peak turns to a trough, the statistics fall and presto, the government claims success.
A small experiment shows how pure chance can produce what looks like evidence in favour of speed cameras, using a simple roll of a die. Before describing it, we had better state our view lest we are labelled extremists of one sort or another. On the statistical evidence, most speed cameras probably do cut accidents, some probably do not, depending where they are. So there is probably a benefit, but the size of that benefit has been greatly exaggerated by some – including the government. And this takes no account of the effect that withdrawing police patrols in favour of using roadside cameras has on accidents caused by other driving offences – drink-driving for example – an effect extremely difficult to estimate. (See also Chapter 7 on risk.)
So, to the experiment. Find a group of people – a typical class size will do but it can be smaller – and ask each person to be a stretch of road, as we did with a gaggle of journalist colleagues. Take your pick: the A42 past Ashby de la Zouche, the B1764 to Doncaster, all perfectly ordinary stretches of road. Next, everyone rolls a die twice and adds the two throws together. This represents the number of accidents on each stretch of road. And, just by chance, some stretches produce higher numbers of accidents than others, as they do in life when brakes or a driver's attention, for example, just happen, by chance, to fail at that moment. In our group of about twenty volunteer journalists we then targeted what appeared to be the accident black-spots, by identifying all those with a score of 10, 11 or 12. To these we gave a photograph of a speed camera, and asked them to roll again, twice. The result? The speed camera photos are instantly effective. None of the high scorers equalled their previous score once they had a photograph of a speed camera.
It might be objected that these were not the real thing, only photographs, and so the experiment proves nothing. But the point is that at some genuine roadside accident sites placing a photograph – or even a pebble – on the pavement would have been as effective as a camera, simply because the rise and fall in the numbers of accidents has been due to chance, just like a six on the roll of a die. Since we have put the cameras (or the photographs of a camera, or the pebble) in a place where the number has just, by chance, been high, it is quite likely, whatever we do, by chance to go down. It looks as if we had something to do with it. In fact, we just caught the wave and took the credit.
Regression to the mean – when a number has reached a peak or trough lately, the next move is likely to be back towards the average – was initially discounted by Department for Transport researchers, despite our challenge to their figures. In the chart opposite, the accident rate goes up and down, as with waves, as in life. Imagine cameras are put in place at time A, when accidents just happened to be high, then it's noticed that they fall to B. This was going to happen anyway, but because the cameras are there it is all too easy to claim that the cameras did it. That is effectively what the Department for Transport did. Then, after claiming for two years a large number of lives saved by cameras, estimates of the benefit were cut sharp
ly in 2006 when, for the first time, some attempt was made to calculate how much of this was due to regression to the mean, and chance was finally given her due.
Even after revision the DfT figures were still unsatisfactory, being the result of a dubious mixture of definitions and some sloppy arithmetic. Figures in the ministerial press release had to be corrected after we challenged them. But in their more modest claims they were at least probably a little closer to the truth.
Of the full fall in the number of people in fatal or serious collisions at speed camera sites, about 60 per cent was now attributed to regression to the mean, and about another 18 per cent to what are called 'trend' effects, namely that the number of accidents was falling everywhere, including where there were no cameras, as a result, for example, of improved road layout and car safety. This left, according to the department's revised figures, about 20 per cent of the apparent benefit of speed cameras genuinely attributable to them, though even this figure is contested.
Figure 5 Accidental success
To observe the tide and not the wave of accidents at speed-camera sites, there are two options: (a) wait until we have seen plenty of waves come and go, which in the case of speed cameras is now believed to be about five years; (b) do not begin measurements only from the peaks, but do plenty of them, and choose points of measurement at random. More simply, we can remember that there is a lot of chance about.
The Tiger That Isn't Page 6