2008 - Bad Science

by Ben Goldacre

  The scientific content of stories—the actual experimental evidence—is brushed over and replaced with didactic statements from authority figures on either side of the debate, which contributes to a pervasive sense that scientific advice is somehow arbitrary, and predicated upon a social role—the ‘expert’—rather than on transparent and readily understandable empirical evidence. Worse than this, other elements are brought into the foreground: political issues, Tony Blair’s refusal to say whether his baby had received the vaccine, mythical narratives, a lionised ‘maverick’ scientist, and emotive appeals from parents.

  A reasonable member of the public, primed with such a compelling battery of human narrative, would be perfectly entitled to regard any expert who claimed MMR was safe as thoughtless and dismissive, especially if that claim came without any apparent supporting evidence.

  The story was also compelling because, like GM food, the MMR story seemed to fit a fairly simple moral template, and one which I myself would subscribe to: big corporations are often dodgy, and politicians are not to be trusted. But it matters whether your political and moral hunches are carried in the right vehicle. Speaking only for myself, I am very wary of drug companies, not because I think all medicine is bad, but because I know they have hidden unflattering data, and because I have seen their promotional material misrepresent science. I also happen to be very wary of GM food—but not because of any inherent flaws in the technology, and not because I think it is uniquely dangerous. Somewhere between splicing in genes for products that will treat haemophilia at one end, and releasing genes for antibiotic resistance into the wild at the other, lies a sensible middle path for the regulation of GM, but there’s nothing desperately remarkable or uniquely dangerous about it as a technology.

  Despite all that, I remain extremely wary of GM for reasons that have nothing to do with the science, simply because it has created a dangerous power shift in agriculture, and ‘terminator seeds’, which die at the end of the season, are a way to increase farmers’ dependency, both nationally and in the developing world, while placing the global food supply in the hands of multinational corporations. If you really want to dig deeper, Monsanto is also very simply an unpleasant company (it made Agent Orange during the Vietnam War, for example).

  Witnessing the blind, seething, thoughtless campaigns against MMR and GM—which minor the infantile train of thought that ‘homeopathy works because the Vioxx side-effects were covered up by Merck’—it’s easy to experience a pervasive sense of lost political opportunities, that somehow all of our valuable indignation about development issues, the role of big money in our society, and frank corporate malpractice, is being diverted away from anywhere it could be valid and useful, and into puerile, mythical fantasies. It strikes me that if you genuinely care about big business, the environment and health, then you’re wasting your time with jokers like Pusztai and Wakefield.

  Science coverage is further crippled, of course, by the fact that the subject can be quite difficult to understand. This in itself can seem like an insult to intelligent people, like journalists, who fancy themselves as able to understand most things, but there has also been an acceleration in complexity in recent times. Fifty years ago you could sketch out a full explanation of how an AM radio worked on the back of a napkin, using basic school-level knowledge of science, and build a crystal set in a classroom which was essentially the same as the one in your car. When your parents were young they could fix their own car, and understand the science behind most of the everyday technology they encountered, but this is no longer the case. Even a geek today would struggle to give an explanation of how his mobile phone works, because technology has become more difficult to understand and explain, and everyday gadgets have taken on a ‘black box’ complexity that can feel sinister, as well as intellectually undermining. The seeds were sown.

  But we should return to the point. If there was little science, then what did appear in all these long stories on MMR? Going back to the 2002 data from the ESRC, only a quarter mentioned Andrew Wakefield, which seems odd, considering he was the cornerstone of the story. This created the erroneous impression that there was a large body of medical opinion which was suspicious of MMR, rather than just one ‘maverick’. Less than a third of broadsheet reports referred to the overwhelming evidence that MMR is safe, and only 11 per cent mentioned that it is regarded as safe in the ninety other countries in which it is used.

  It was rare to find much discussion of the evidence at all, as it was considered too complicated, and when doctors tried to explain it they were frequently shouted down, or worse still, their explanations were condensed into bland statements that ‘science had shown’ there was nothing to worry about. This uninformative dismissal was pitted against the emotive concerns of distressed parents.

  As 2002 wore on, things got really strange. Some newspapers, such as the Daily Mail and the Daily Telegraph, made MMR the focus of a massive political campaign, and the beatification of Wakefield reached a kind of fever pitch. Lorraine Fraser had an exclusive interview with him in the Telegraph in which he was described as ‘a champion of patients who feel their fears have been ignored’. She wrote a dozen similar articles over the next year (and her reward came when she was named British Press Awards Health Writer of the Year 2002, a gong I do not myself expect to receive).

  Justine Picardie did a lavish photo feature on Wakefield, his house and his family for the Telegraph Saturday magazine. Andy is, she tells us, ‘a handsome, glossy-haired hero to families of autistic children’. How are the family? ‘A likeable, lively family, the kind you would be happy to have as friends, pitted against mysterious forces who have planted bugging devices and have stolen patients’ records in ‘apparently inexplicable’ burglaries.’ She fantasises—and I absolutely promise you I’m not making this up—about a Hollywood depiction of Wakefield’s heroic struggle, with Russell Crowe playing the lead ‘opposite Julia Roberts as a feisty single mother fighting for justice for her child’.

  The evidence on MMR

  So what is the evidence on the safety of MMR?

  There are a number of ways to approach the evidence on the safety of a given intervention, depending on how much attention you have to give. The simplest approach is to pick an arbitrary authority figure: a doctor, perhaps, although this seems not to be appealing (in surveys people say they trust doctors the most, and journalists the least: this shows the flaw in that kind of survey).

  You could take another, larger authority at face value, if there is one that suits you. The Institute of Medicine, the Royal Colleges, the NHS, and more, all came out in support of MMR, but this was apparently not sufficient to convince. You could offer information: an NHS website at mmrthefacts.nhs.uk started with the phrase ‘MMR is safe’ (literally), and allowed the reader to drill down to the detail of individual studies.*

  ≡ Whether you buy the DoH phrase ‘MMR is safe’ depends on what you decide you mean by ‘safe’. Is flying safe? Is your washing machine safe? What are you sitting on? Is that safe? You can obsess over the idea that philosophically nothing can ever be shown to be 100 per cent safe—and many will—but you would be arguing about a fairly meaningless and uncommon definition of the word.

  But that too did little to stem the tide. Once a scare is running, perhaps every refutation can seem like an admission of guilt, drawing attention to the scare.

  The Cochrane Collaboration is as blemishless as they come, and it has done a systematic review of the literature on MMR, concluding that there was no evidence that it is unsafe (although it didn’t appear until 2005). This reviewed the data the media had systematically ignored: what was in it?

  If we are to maintain the moral high ground, there are a few things we need to understand about evidence. Firstly, there is no single golden study which proves that MMR is safe (although the evidence to say it is dangerous was exceptionally poor). There is, for example, no randomised controlled trial. We are presented instead with a huge morass of data, from a number of
different studies, all of which are flawed in their own idiosyncratic ways for reasons of cost, competence and so on. A common problem with applying old data to new questions is that these papers and datasets might have a lot of useful information, which was collected very competently to answer the questions which the researchers were interested in at the time, but which isn’t perfect for your needs. It’s just, perhaps, pretty good.

  Smeeth et al., for example, did something called a ‘case-control’ study, using the GP Research Database. This is a common type of study, where you take a bunch of people with the condition you’re looking at (‘autism’), and a bunch of people without it, then look to see if there is any difference in how much each group was exposed to the thing you think might be causing the condition (‘MMR’).

  If you care who paid for the study—and I hope you’ve become a bit more sophisticated than that by now—it was funded by the Medical Research Council. They found around 1,300 people with autism, and then got some ‘controls’: random people who did not have autism, but with the same age, sex, and general practice. Then they looked to see if vaccination was any more common in the people with autism, or the controls, and found no difference between the two groups. The same researchers also did a systematic review of similar studies in the United States and Scandinavia, and again, pooling the data, found no link between MMR and autism.

  There is a practical problem with this kind of research, of course, which I would hope you might spot: most people do get the MMR vaccine, so the individuals you’re measuring who didn’t get the vaccine might be unusual in other ways—perhaps their parents have refused the vaccine for ideological or cultural reasons, or the child has a pre-existing physical health problem—and those factors might themselves be related to autism. There’s little you can do in terms of study design about this potential ‘confounding variable’, because as we said, you’re not likely to do a randomised controlled trial in which you randomly don’t give children vaccines: you just throw the result into the pot with the rest of the information, in order to reach your verdict. As it happens, Smeeth et al. went to great lengths to make sure their controls were representative. If you like, you can read the paper and decide if you agree.

  So ‘Smeeth’ was a ‘case-control study’, where you compare groups which had the outcome or not, and look at how common the exposure was in each group. In Denmark, Madsen et al. did the opposite kind of study, called a ‘cohort study’: you compare groups which had the exposure or not, in order to see whether there is any variation in the outcome. In this specific case, then, you take two groups of people, who either had MMR or didn’t, and then check later to see if the rate of autism is any different between the two groups.

  This study was big—very big—and included all the children born in Denmark between January 1991 and December 1998. In Denmark there is a system of unique personal identification numbers, linked to vaccination registers and information about the diagnosis of autism, which made it possible to chase up almost all the children in the study. This was a pretty impressive achievement, since there were 440,655 children who were vaccinated, and 96,648 who were unvaccinated. No difference was found between vaccinated and unvaccinated children, in the rates of autism or autistic spectrum disorders, and no association between development of autism and age at vaccination.

  Anti-MMR campaigners have responded to this work by saying that only a small number of children are harmed by the vaccine, which seems to be inconsistent with their claims that MMR is responsible for a massive upswing in diagnoses of autism. In any case, if a vaccine caused an adverse reaction in a very small number of people, that would be no surprise—it would be no different from any other medical intervention (or, arguably, any human activity), and there would be, surely, no story.

  As with all studies, there are problems with this huge study. The follow-up of diagnostic records ends one year (31 December 1999) after the last day of admission to the cohort: so, because autism comes on after the age of one year, the children born later in the cohort would be unlikely to have shown up with autism by the end of the follow-up period. But this is nagged up in the study, and you can decide whether you think it undermines its overall findings. I don’t think it’s much of a problem. That’s my verdict, and I think you might agree that it’s not a particularly foolish one. It did run from January 1991, after all.

  This is the kind of evidence you will find in the Cochrane review, which found, very simply, that ‘existing evidence on the safety and effectiveness of MMR vaccine supports current policies of mass immunisation aimed at global measles eradication in order to reduce morbidity and mortality associated with mumps and rubella’.

  It also contained multiple criticisms of the evidence it reviewed, which, bizarrely, has been seized upon by various commentators to claim that there was some kind of stitch-up. The review was heading towards a conclusion that MMR was risky, they say, if you read the content, but then, out of nowhere, it produced a reassuring conclusion, doubtless because of hidden political pressure.

  The Daily Mail’s Melanie Phillips, a leading light of the anti-vaccination movement, was outraged by what she thought she had found: ‘It said that no fewer than nine of the most celebrated studies that have been used against [Andrew Wakefield] were unreliable in the way they were constructed.’ Of course it did. I’m amazed it wasn’t more. Cochrane reviews are intended to criticise papers.

  Scientific ‘evidence’ in the media

  But the newspapers in 2002 had more than just worried parents. There was a smattering of science to keep things going: you will remember computer-generated imagery of viruses and gut walls, perhaps, and stories about laboratory findings. Why have I not mentioned those?

  For one thing, these important scientific findings were being reported in newspapers and magazines, and at meetings, in fact anywhere except proper academic journals where they could be read and carefully appraised. In May, for example, Wakefield ‘exclusively revealed’ that ‘more than 95 per cent of those who had the virus in their gut had MMR as their only documented exposure to measles’. He doesn’t appear to have revealed this in a peer-reviewed academic journal, but in a weekend colour supplement.

  Other people started popping up all over the place, claiming to have made some great finding, but never publishing their research in proper, peer-reviewed academic journals. A pharmacist in Sunderland called Mr Paul Shattock was reported on the Today programme, and in several national newspapers, to have identified a distinct subgroup of children with autism resulting from MMR. Mr Shattock is very active on anti-immunisation websites, but he still doesn’t seem to have got round to publishing this important work years later, even though the Medical Research Council suggested in 2002 that he should ‘publish his research and come forward to the MRC with positive proposals’.

  Meanwhile Dr Arthur Krigsman, paediatric gastrointestinal consultant working in the New York area, was telling hearings in Washington that he had made all kinds of interesting findings in the bowels of autistic children, using endoscopes. This was lavishly reported in the media. Here is the Daily Telegraph:

  Scientists in America have reported the first independent corroboration of the research findings of Dr Andrew Wakefield. Dr Krigsman’s discovery is significant because it independently supports Dr Wakefield’s conclusion that a previously unidentified and devastating combination of bowel and brain disease is afflicting young children—a claim that the Department of Health has dismissed as ‘bad science’.

  To the best of my knowledge—and I’m pretty good at searching for this stuff- Krigsman’s new scientific research findings which corroborate Andrew Wakefield’s have never been published in an academic journal: certainly there is no trace of them on Pubmed, the index of nearly all medical academic articles.

  In case the reason why this is important has not sunk in, let me explain again. If you visit the premises of the Royal Society in London, you’ll see its motto proudly on display: ‘Nullius in verba ‘—‘On
the word of no one’. What I like to imagine this refers to, in my geeky way, is the importance of publishing proper scientific papers if you want people to pay attention to your work. Dr Arthur Krigsman has been claiming for years now that he has found evidence linking MMR to autism and bowel disease. Since he hasn’t published his findings, he can claim them until he’s blue in the face, because until we can see exactly what he did, we can’t see what flaws there may be in his methods. Maybe he didn’t select the subjects properly. Maybe he measured the wrong things. If he doesn’t write it up formally, we can never know, because that is what scientists do: write papers, and pull them apart to see if their findings are robust.

  Krigsman and others’ failures to publish in peer-reviewed academic journals weren’t isolated incidents. In fact it’s still going on, years later. In 2006, exactly the same thing was happening again. ‘US Scientists Back Autism Link to MMR’, squealed the Telegraph. ‘Scientists Fear MMR Link to Autism’, roared the Mail. ‘US Study Supports Claims of MMR Link to Autism’, croaked The Times a day later.

  What was this frightening new data? These scare stories were based on a poster presentation, at a conference yet to occur, on research not yet completed, by a man with a track record of announcing research that never subsequently appears in an academic journal. In fact, astonishingly, four years later, it was Dr Arthur Krigsman again. The story this time was different: he had found genetic material (RNA) from vaccine-strain measles virus in some gut samples from children with autism and bowel problems. If true, this would fit with Wakefield’s theory, which by 2006 was lying in tatters. We might also mention that Wakefield and Krigsman are doctors together at Thoughtful House, a private autism clinic in the USA offering eccentric treatments for developmental disorders.