by Ben Goldacre
Over There! An Eight-Mile-High Distraction Made of Posh Chocolate!
Guardian, 1 August 2009
This week the Food Standards Agency published two review papers showing that organic food is no better than normal food, in terms of composition, or health benefits. The Soil Association’s response has been swift, and it has been given prominent, blanket right of reply throughout the media. That is testament to the lobbying power of a £2 billion industry, and the cultural values of journalists. I don’t care about organic food, but I am interested in bad arguments. The Soil Association has made three.
Firstly, it says that the important issue with organic food is not the personal health benefit (there is none to be found in this data), but rather the benefit to the environment. This strategy – ‘Don’t talk about that, talk about this’ – is a popular one, but it is cheap: we can talk separately about the environmental issues with organic food, but right now we are talking about the health effects.
Secondly, it says that the health benefits of organic food are related to pesticides, and cannot be measured by the evidence that has been identified and summarised in the FSA paper. This, once more, is gamesmanship. There was no evidence of health benefits to individuals. Possibly the Soil Association is proposing that there are health benefits which somehow cannot be measured: it is hard to disentangle the health benefits of eating organic food from other beneficial features of people’s lives, for example, when you measure their health in a thirty-year study. In this case it is expressing a position of faith, not evidence. Or it is proposing that there are health benefits which could be measured, but have not been measured yet. In that case, again, this is faith rather than evidence, but it could start recruiting researchers now, using some small portion of its industry’s £2 billion revenue to investigate these beliefs with fair tests.
Lastly, like many vitamin-pill peddlers and pharmaceutical companies before it, the Soil Association seeks to undermine the public’s understanding of what a ‘systematic review’ is. It says that the report has deliberately excluded evidence to produce the answer that organic food is no better. The accusation is one of cherry-picking, and it is hard to see how it can be valid in the kind of studies that have been published here. These are ‘systematic reviews’: before you begin collecting papers to include, you specify how you will search for evidence, what databases you will use, what types of studies you will use, how you will grade the quality of the evidence (to see if it was a ‘fair test’), and so on.
What does the Soil Association think these systematic reviews have ignored? As an example, from its press release, the industry body is ‘disappointed that the FSA failed to include the results of a major European Union-funded study involving thirty-one research and university institutes and the publication, so far, of more than one hundred scientific papers, at a cost of 18 million Euros, which ended in April this year’. It gave the link to www.qlif.org.
On this website, you will find the QLIF list of 120 papers. Almost all are irrelevant. The first fourteen are on ‘consumer expectations and attitudes’, which are correctly not included in a systematic review of the evidence on food composition and health. Then there are twenty-two on ‘effects of production methods’. Here you might expect to find more relevant research, but no.
The first paper (‘The effect of medium term feeding with organic, low input and conventional diet on selected immune parameters in rat’), while interesting, will plainly not be relevant to a systematic review on nutrient content. The same is true of the next paper, ‘Salmonella Infection Level in Danish Indoor and Outdoor Pig Production Systems measured by Antibodies in Meat Juice and Faecal Shedding on-farm and at Slaughter’: you might love its results, you might hate them; either way, it doesn’t matter. This paper is simply not relevant to a review on nutrient content.
What’s more, the overwhelming majority of these studies are unpublished conference papers, and some are just brief descriptions of the fact that somebody made an oral presentation at a meeting. The systematic review correctly looked only at good-quality data published in peer-reviewed academic journals, and with good reason: we know that conference papers are unreliable sources of information, that they often change between conference and publication, and that they are often never published at all.
As so often, this is about transparency, which is ultimately the only source of authority in science: we want the methods and results of scientific research to be formally presented, and accessible by all, so that we can see what was done, and what they found. If a government report on anything relies substantially on unpublished and inaccessible research, then we are correctly concerned. In fact, just two weeks ago this column discussed how the key piece of evidence presented by the Home Office to justify retaining DNA from innocent people who have only ever been arrested and then released was an incompetently presented piece of unpublished, incomplete research.
Systematic reviews give you transparency, because from the very outset everyone is honest and open about what kinds of studies they will and won’t include. When organisations like the Soil Association greet the publication of a systematic review with accusations of cherry-picking, they’re not just wrong, they’re undermining the public’s understanding of one of the most important and simple new ideas from the past three decades of science.
That annoys me, so let me share a prejudice. Ultimately, when people talk about the health benefits of organic food, they’re not really talking about the health benefits of organic food. Wealthy society’s big love for organic farming is about something very different: a bundle of legitimate concerns about unchecked capitalism in our food supply, battery farming, corruptible regulators, and reckless destruction of the environment, where the producers’ costs do not reflect the true full costs of their activities to society, to name just a few. Every one of these problems deserves our full, individual attention.
But magic spells will not work. We cannot eradicate deceit from the pharmaceutical industry by buying homeopathic sugar pills; and we cannot solve the problems of unchecked capitalism in industrial food production by giving money to the £2 billion organic food industry in exchange for the occasional posh carrot.
As Far as I Understand Thinktanks …
Guardian, 7 June 2008
There has been a frightening decline in the quality of maths in reports complaining about the frightening decline in the quality of maths in Britain. ‘The Value of Mathematics’, by thinktank Reform, has received large quantities of flattering media coverage this week from The Times, the Telegraph, and even scored a second puff in the Guardian from Professor Marcus du Sautoy, Oxford’s Professor of the Public Understanding of Science. Their argument is simple: there is less maths around; people think it’s cool to be bad at sums; we suffer economically; these are bad things.
Here is a key, early, factual claim from Reform’s report: ‘About 40 per cent of mathematics graduates enter financial services.’ This, they say, is a good thing. Do we believe the number? The report references it to the front page of a website called prospects.ac.uk, which is a pretty big website. Chasing through the pages there, you will find ‘What Do Graduates Do?’, and then the maths page. There were 4,070 maths graduates in their sampling frame for the year 2006. Only 2,010 of those, however, are in UK employment (1.5 per cent are working abroad, and the rest are studying for a higher degree, or a teaching qualification, or are unemployed, or unavailable for employment, and so on).
Of those 2,010 – not 4,070 – 37.9 per cent are working as ‘Business and Financial Professionals and Associate Professionals’. So if we use maths: 2,010 × 0.379 = 761.79, and that divided by 4,070 gives us 0.1871, but let’s round up like the angry maths profs did. About 20 per cent of maths graduates enter financial services. Not 40 per cent. For a group of people complaining about the substitution of woolly modern notions like ‘relevance’ and ‘applied maths’ in place of high-end mathematical techniques, these maths profs aren’t very good at arithmetic.
/>
They’re also not very good at basic applied maths. For example, they argue that if we simply had more people around with maths knowledge, then there would automatically be more (and more lucrative) jobs requiring that knowledge, which our new maths graduates would instantly take up. I’m not an economist, but I’m not sure labour markets work like that: there are lots of men in the north of England who are very good at mining, and nobody in a hurry to open new pits.
Unfortunately, in any case, even this aspect of the report seems to be marred by simple errors in applied arithmetic. The thinktank is worried that the loss of A-level mathematicians has resulted in lost earnings for the economy. If the number of maths candidates had remained constant, they say, there would have been an additional 430,700 over the period 1989 to 2007. In the adjacent table they say 430,031, but that’s the least of our worries. They go on to reason like this: ‘Each of these students would have earned an additional £3,080 per year due to the market premium on A-level mathematics, equating to £136,000 over their lifetime. The total gain to the economy over the period would have been over £9 billion.’
We’ll put aside the fact that the BBC said ‘A maths A-level puts on average an extra £10,000 a year on a salary [not £3,080], says Reform,’ because I can’t get £9 billion for that period with those numbers (I get £12 billion, assuming a linear decline). Even if I could, Reform are making assumptions that are hard to blindly accept: in particular, will the extra earnings for people with maths A-level really still hold, if more people have maths A-level? We could go on.
I’m happy to agree that maths is economically useful, that some people might want to avoid difficult school subjects, and that humanities graduates who think maths is uncool are bores. What I would like is someone who can be bothered to sit down and reinforce my prejudices without perpetrating crass errors of over-interpretation and getting the basic arithmetic wrong. I’ve never fully seen the point of them, but until now, I assumed that’s what thinktanks and Professors of the Public Understanding of Science are there for.
Meaningful Debates Need Clear Information
Guardian, 27 October 2007
Where do all those numbers in the newspapers come from? The Commons Committee on Science and Technology is taking evidence on ‘scientific developments relating to the Abortion Act 1967’.
Scientific and medical expert bodies giving evidence say that survival in births below the twenty-fourth week of pregnancy has not significantly improved since the 1990s, when it was only 10–20 per cent. But one expert, a Professor of Neonatal Medicine, says survival at twenty-two and twenty-three weeks has improved. In fact, he says survival rates in this group can be phenomenally high: 42 per cent of children born at twenty-three weeks at some top specialist centres. He has been quoted widely: in the Independent and the Telegraph, on Channel 4 and Newsnight, by Tory MPs, and so on. The figure has a life of its own.
In the media, you get one expert saying one thing, and another saying something else. Who do you believe? The devil is in the detail. One option is to examine the messenger. John Wyatt is a member of the Christian Medical Fellowship. He didn’t declare that when he went to give evidence. You don’t have to. He did declare it when asked.
Prof Wyatt has relevant research experience, but there were half a dozen other medics without any relevant background who submitted evidence (or their view of it) to the committee who, when asked if they had anything to declare, did mention membership of Christian or evangelical groups with an established position on abortion. I don’t care for an argument that rests on competing ideologies, so let’s look at Prof Wyatt’s evidence: because it has been hugely reported, and it goes against the evidence from a huge study called Epicure.
Epicure contains all of the data for every premature birth in England over the course of a year: one snapshot was taken in 1995, and one in 2006. Overall, it shows a modest improvement in survival for births at twenty-four weeks, but no significant improvement in the 10–20 per cent rate for births at twenty-two and twenty-three weeks.
For the next bit, you need to remember one simple piece of primary school maths. In the figure 3/20, 3 is the numerator and 20 is the denominator. If you have three survivors for every twenty births, then 15 per cent survive. For Epicure, the numerator is survival to discharge from hospital, and the denominator is all births where there is a sign of life, carefully defined.
There are two ways you could get a higher survival percentage. One would be a genuine increase in the number of babies surviving, an increase in the numerator: eight out of twenty live births survive, 40 per cent. But you could also see an increase in the survival percentage by changing the denominator. Let’s say, instead of counting as your denominator ‘all births where there is a sign of life in the delivery room’, you counted ‘all babies admitted to neonatal intensive care’. Now, that’s a different kettle of fish altogether. To be admitted to neonatal ICU, the doctors have to think you’ve got a chance. Often you have to be transferred from another hospital in an ambulance, and for that you really do have to be more well. Therefore, if your denominator is ‘neonatal ICU admissions’, your survival rates will be higher, but you are not comparing like with like. That may partly explain Prof Wyatt’s figure for a very high survival rate in twenty-three-week babies. But it’s not clear.
First, in his written evidence he said that the data was from a ‘prospectively defined’ study (where the researchers say in advance what they plan to collect). Then he was asked in the committee, when giving his oral evidence: ‘What was the denominator for that? Was that … 42 per cent survival at twenty-three weeks of all babies showing signs of life in the delivery room, or was it a proportion of those admitted to neonatal intensive care directly or by transfer?’ Prof Wyatt replied: ‘The denominator was all babies born alive in the labour ward in the hospital at UCL [University College London].’ This later turned out not to be true.
Then he was asked to send the reference for the claim. He did so. It was merely an abstract for an academic conference presentation three years ago. It did not contain the figures he was quoting. He then said he had done the raw figures on a spreadsheet, especially for the committee – bespoke, if you will – and sent them in. They are entered into the record as a memo, on 18 October 2007. They show new, different, but broadly comparable figures: 50 per cent survive at twenty-two weeks, then down to 46 per cent at twenty-three weeks, then up to 82 per cent at twenty-four weeks, then down again to 77 per cent at twenty-five weeks. (That bouncing around is because the raw numbers are so small that there is a lot of random noise.)
And the denominator? Prof Wyatt is clear: ‘I have provided the numbers and percentage of infants born alive at University College London Hospitals who survived to one year of age.’ The committee asked for clarification of this. Finally, on 23 October, another memo arrived from Prof Wyatt, entered into the record where all can read it. For the widely quoted 42 per cent survival rate at twenty-three weeks, Prof Wyatt admitted that the denominator was, in fact, all babies admitted to the neonatal intensive care unit. But in his new special analysis, giving this new ‘46 per cent survive at twenty-three weeks’ figure, the figures in the previous paragraph, he claimed the denominator was ‘all live births’. Has he undone a prospectively designed study, and retrospectively redesigned it? Or is this now a completely different source of data to the original reference?
I cannot blame Prof Wyatt for this, but his figure has taken on a life of its own. There may have been yet another mistake here, about the denominator. I don’t know. I’m quite prepared to believe that UCL may have unusually good results. But science is about clarity and transparency, especially for public policy. You need to be very clear on things like: what do you define as a ‘live birth’, how do you decide on what gestational age was, and so on. Even if this data stands up eventually, right now it is non-peer-reviewed, unpublished, utterly chaotic, changeable, personal communication of data, from 1996 to 2000, with no clear source, and no informatio
n about how it was collected or analysed. That would be fine if it hadn’t suddenly become central to the debate on abortion.
Minority Retort
Guardian, 3 November 2007
Parliamentary select committees are among the few places where you can see politicians sitting down and doing the kind of thing you’d actually want them to do, like thinking carefully about policy. This week the Science and Technology Committee delivered its report on scientific developments relating to the Abortion Act. This is, entirely for free, a fantastic piece of popular science writing on epidemiology. In particular, it’s a masterclass on spotting dodgy statistics, which is exactly what the committee received from anti-abortion activists.
Here is one example, on the question of abortion and breast cancer risk, in which a parliamentary document explains the importance of choosing the correct control group.
Dr Richards told us that ‘if you compare women who keep their pregnancy with those who have an induced abortion, those who have an induced abortion are more likely to get breast cancer later on’. This is the comparative group that Dr Brind favours and the result is expected, since carrying a first pregnancy to birth is protective against breast cancer. However, if you look at the rates of cancer between women who have had an abortion and those who have not had children, the effect disappears.