2008 - Bad Science

by Ben Goldacre

  The Telegraph went on to explain that Krigsman’s most recent unpublished claim was replicating similar work from 1998 by Dr Andrew Wakefield, and from 2002 by Professor John O’Leary. This was, to say the least, a mis-statement. There is no work from 1998 by Wakefield which fits the Telegraph’s claim—at least not in PubMed that I can find. I suspect the newspaper was confused about the infamous Lancet paper on MMR, which by 2004 had already been partially retracted.

  There are, however, two papers suggesting that traces of genetic material from the measles virus have been found in children. They have received a mountain of media coverage over half a decade, and yet the media have remained studiously silent on the published evidence suggesting that they were false positives, as we will now see.

  One is from Kawashima et al. in 2002, also featuring Wakefield as an author, in which it is claimed that genetic material from measles vaccine was found in blood cells. Doubt is cast on this both by attempts to replicate it, showing where the false positives probably appeared, and by the testimony of Nick Chadwick, the PhD student whose work we described above. Even Andrew Wakefield himself no longer relies on this paper.

  The other is O’Leary’s paper from 2002, also featuring Wakefield as an author, which produced evidence of measles RNA in tissue samples from children. Futher experiments, again, have illustrated where the false positives seem to have arisen, and in 2004, when Professor Stephen Bustin was examining the evidence for the legal aid case, he explained how he established to his satisfaction—during a visit to the O’Leary lab—that these were false positives due to contamination and inadequate experimental methods. He has shown, firstly, that there were no ‘controls’ to check for false positives (contamination is a huge risk when you are looking for minuscule traces of genetic material, so you generally run ‘blank’ samples to make sure they do come out blank); he found calibration problems with the machines; problems with log books; and worse. He expanded on this at enormous length in a US court case on autism and vaccines in 2006. You can read his detailed explanation in full online. To my astonishment not one journalist in the UK has ever bothered to report it.

  Both of these papers claiming to show a link received blanket media coverage at the time, as did Krigsman’s claims.

  What they didn’t tell you

  In the May 2006 issue of the Journal of Medical Virology there was a very similar study to the one described by Krigsman, only this one had actually been published, by Afzal et al. It looked for measles RNA in children with regressive autism after MMR vaccination, much like the unpublished Krigsman study, and it used tools so powerful they could detect measles RNA down to single-figure copy numbers. It found no evidence of the magic vaccine-strain measles RNA to implicate MMR. Perhaps because of that unfrightening result, the study was loudly ignored by the press.

  Because it has been published in full, I can read it, and pick holes in it, and I am more than happy to do so: because science is about critiquing openly published data and methodologies, rather than press-released chimeras, and in the real world all studies have some flaws, to a greater or lesser extent. Often they are practical ones: here, for example, the researchers couldn’t get hold of the tissue they ideally would have used, because they could not get ethics committee approval for intrusive procedures like lumbar punctures and gut biopsies on children (Wakefield did manage to obtain such samples, but he is, we should remember, currently going through a GMC professional conduct hearing over the issue).

  Surely they could have borrowed some existing samples, from children said to be damaged by vaccines? You’d have thought so. They report in the paper that they tried to ask anti-MMR researchers—if that’s not an unfair term—whether they could borrow some of their tissue samples to work on. They were ignored.*

  ≡ ‘The groups of investigators that either had access to original autism specimens or investigated them later for measles virus detection were invited to take part in the study but failed to respond. Similarly, it was not possible to obtain clinical specimens of autism cases from these investigators for independent investigations.’

  Afzal et al. was not reported in the media, anywhere at all, except by me, in my column.

  This is not an isolated case. Another major paper was published in the leading academic journal Pediatrics a few months later—to complete media silence—again suggesting very strongly that the earlier results from Kawashima and O’Leary were in error, and false positives. D’Souza et al. replicated the earlier experiments very closely, and in some respects more carefully: most importantly, it traced out the possible routes by which a false positive could have occurred, and made some astonishing findings.

  False positives are common in PCR, because it works by using enzymes to replicate RNA, so you start with a small amount in your sample, which is then ‘amplified up’, copied over and over again, until you have enough to measure and work with. Beginning with a single molecule of genetic material, PCR can generate 100 billion similar molecules in an afternoon. Because of this, the PCR process is exquisitely sensitive to contamination—as numerous innocent people languishing in jail could tell you—so you have to be very careful, and clean up as you go.

  As well as raising concerns about contamination, D’Souza also found that the O’Leary method might have accidentally amplified the wrong bits of RNA.

  Let’s be clear: this is absolutely not about criticising individual researchers. Techniques move on, results are sometimes not replicable, and not all double-checking is practical (although Bustin’s testimony is that standards in the O’Leary lab were problematic). But what is striking is that the media rabidly picked up on the original frightening data, and then completely ignored the new reassuring data. This study by D’Souza, like Afzal before it, was unanimously ignored by the media. It was covered, by my count, in: my column; one Reuters piece which was picked up by nobody; and one post on the lead researcher’s boyfriend’s blog (where he talked about how proud he was of his girlfriend). Nowhere else.*

  ≡ In 2008, just as this chapter was being put to bed, some journalists deigned—miraculously—to cover a PCR experiment with a negative finding. It was misreported as the definitive refutation of the entire MMR-autism hypothesis. This was a childish overstatement, and that doesn’t help anyone either. I am not hard to please.

  You could say, very reasonably, that this is all very much par for the course: newspapers report the news, and it’s not very interesting if a piece of research comes out saying something is safe. But I would argue—perhaps sanctimoniously—that the media have a special responsibility in this case, because they themselves demanded ‘more research’, and moreover because at the very same time that they were ignoring properly conducted and fully published negative findings, they were talking up scary findings from an unpublished study by Krigsman, a man with a track record of making scary claims which remain unpublished.

  MMR is not an isolated case in this regard. You might remember the scare stories about mercury fillings from the past two decades: they come around every few years, usually accompanied by a personal anecdote in which fatigue, dizziness and headaches are all vanquished following the removal of the fillings by one visionary dentist. Traditionally these stories conclude with a suggestion that the dental establishment may well be covering up the truth about mercury, and a demand for more research into its safety.

  The first large-scale randomised control trials on the safety of mercury fillings were published recently, and if you were waiting to see these hotly anticipated results, personally demanded by journalists on innumerable newspapers, you’d be out of luck, because they were reported nowhere. Nowhere. A study of more than 1,000 children, where some were given mercury fillings and some mercury-free fillings, measuring kidney function and neurodevelopmental outcomes like memory, coordination, nerve conduction, IQ and so on over several years. It was a well-conducted study. There were no significant differences between the two groups. That’s worth knowing about if you’ve ever been sca
red by the media’s reports on mercury fillings—and by God, you’d have been scared.

  Panorama featured a particularly chilling documentary in 1994 called The Poison in Your Mouth. It opened with dramatic footage of men in full protective gear rolling barrels of mercury around. I’m not giving you the definitive last word on mercury here. But I think we can safely assume there is no Panorama documentary in the pipeline covering the startling new research data suggesting that mercury fillings may not be harmful after all. In some respects this is just one more illustration of how unreliable intuition can be in assessing risks like those presented with a vaccine: not only is it a flawed strategy for this kind of numerical assessment, on outcomes which are too rare for one person to collect meaningful data on them in their personal journey through life; but the information you are fed by the media about the wider population is ludicrously, outrageously, criminally crooked. So at the end of all this, what has the British news media establishment achieved?

  Old diseases return

  It’s hardly surprising that the MMR vaccination rate has fallen from 92 per cent in 1996 to 73 per cent today. In some parts of London it’s down to 60 per cent, and figures from 2004-05 showed that in Westminster only 38 per cent of children had both jabs by the age of five.*

  ≡ Not 11.7 per cent as claimed in the Telegraph and the Daily Mail in February and June 2006.

  It is difficult to imagine what could be driving this, if not a brilliantly successful and well-coordinated media anti-MMR campaign, which pitched emotion and hysteria against scientific evidence. Because people listen to journalists: this has been demonstrated repeatedly, and not just with the kinds of stories in this book.

  A 2005 study in the Medical Journal of Australia looked at mammogram bookings, and found that during the peak media coverage of Kylie Minogue’s breast cancer, bookings rose by 40 per cent. The increase among previously unscreened women in the forty-to-sixty-nine-year age group was 101 per cent. These surges were unprecedented. And I’m not cherry-picking: a systematic review from the Cochrane Collaboration found five studies looking at the use of specific health interventions before and after media coverage of specific stories, and each found that favourable publicity was associated with greater use, and unfavourable coverage with lower use.

  It’s not just the public: medical practice is influenced by the media too, and so are academics. A mischievous paper from the New England Journal of Medicine in 1991 showed that if a study was covered by the New York Times, it was significantly more likely to be cited by other academic papers. Having come this far, you are probably unpicking this study already. Was coverage in the New York Times just a surrogate marker for the importance of the research? History provided the researchers with a control group to compare their results against: for three months, large parts of the paper went on strike, and while the journalists did produce an ‘edition of record’, this newspaper was never actually printed. They wrote stories about academic research, using the same criteria to judge importance that they always had, but the research they wrote about in articles which never saw the light of day saw no increase in citations.

  People read newspapers. Despite everything we think we know, their contents seep in, we believe them to be true, and we act upon them, which makes it all the more tragic that their contents are so routinely flawed. Am I extrapolating unfairly from the extreme examples in this book? Perhaps not. In 2008 Gary Schwitzer, an ex-journalist who now works on quantitative studies of the media, published an analysis of five hundred health articles covering treatments from mainstream newspapers in the US. Only 35 per cent of stories were rated satisfactory for whether the journalist had ‘discussed the study methodology and the quality of the evidence’ (because in the media, as we have seen repeatedly in this book, science is about absolute truth statements from arbitrary authority figures in white coats, rather than clear descriptions of studies, and the reasons why people draw conclusions from them). Only 28 per cent adequately covered benefits, and only 33 per cent adequately covered harms. Articles routinely failed to give any useful quantitative information in absolute terms, preferring unhelpful eye-catchers like ‘50 per cent higher’ instead.

  In fact there have been systematic quantitative surveys of the accuracy of health coverage in Canada, Australia and America—I’m trying to get one off the ground in the UK—and the results have been universally unimpressive. It seems to me that the state of health coverage in the UK could well be a serious public health issue.

  Meanwhile, the incidence of two of the three diseases covered by MMR is now increasing very impressively. We have the highest number of measles cases in England and Wales since current surveillance methods began in 1995, with cases occurring mostly in children who had not been adequately vaccinated: 971 confirmed cases were reported in 2007 (mostly associated with prolonged outbreaks in travelling and religious communities, where vaccine uptake has been historically low), after 740 cases in 2006 (and the first death since 1992). Seventy-three per cent of cases were in the South–East, and most of those were in London.

  Mumps began rising again in 1999, after many years of cases in only double figures: by 2005 the United Kingdom had a mumps epidemic, with around 5,000 notifications in January alone.

  A lot of people who campaign against vaccines like to pretend that they don’t do much good, and that the diseases they protect against were never very serious anyway. I don’t want to force anyone to have their child vaccinated, but equally I don’t think anyone is helped by misleading information. By contrast with the unlikely event of autism being associated with MMR, the risks from measles, though small, are real and quantifiable. The Peckham Report on immunisation policy, published shortly after the introduction of the MMR vaccine, surveyed the recent experience of measles in Western countries and estimated that for every 1,000 cases notified, there would be 0.2 deaths, ten hospital admissions, ten neurological complications and forty respiratory complications. These estimates have been borne out in recent minor epidemics in the Netherlands (1999: 2,300 cases in a community philosophically opposed to vaccination, three deaths), Ireland (2000: 1,200 cases, three deaths) and Italy (2002: three deaths). It’s worth noting that plenty of these deaths were in previously healthy children, in developed countries, with good healthcare systems.

  Though mumps is rarely fatal, it’s an unpleasant disease with unpleasant complications (including meningitis, pancreatitis and sterility). Congenital rubella syndrome has become increasingly rare since the introduction of MMR, but causes profound disabilities including deafness, autism, blindness and mental handicap, resulting from damage to the foetus during early pregnancy.

  The other thing you will hear a lot is that vaccines don’t make much difference anyway, because all the advances in health and life expectancy have been due to improvements in public health for a wide range of other reasons. As someone with a particular interest in epidemiology and public health, I find this suggestion flattering; and there is absolutely no doubt that deaths from measles began to fall over the whole of the past century for all kinds of reasons, many of them social and political as well as medical: better nutrition, better access to good medical care, antibiotics, less crowded living conditions, improved sanitation, and so on.

  Life expectancy in general has soared over the past century, and it’s easy to forget just how phenomenal this change has been. In 1901, males born in the UK could expect to live to forty-five, and females to forty-nine. By 2004, life expectancy at birth had risen to seventy-seven for men, and eighty-one for women (although of course much of the change is due to reductions in infant mortality).

  So we are living longer, and vaccines are clearly not the only reason why. No single change is the reason why. Measles incidence dropped hugely over the preceding century, but you would have to work fairly hard to persuade yourself that vaccines had no impact on that. Here, for example, is a graph showing the reported incidence of measles from 1950 to 2000 in the United States.

  For thos
e who think that single vaccines for the components of MMR are a good idea, you’ll notice that these have been around since the 1970s, but that a concerted programme of vaccination—and the concerted programme of giving all three vaccinations in one go as MMR—is fairly clearly associated in time with a further (and actually rather definitive) drop in the rate of measles cases.

  The same is true for mumps:

  While we’re thinking about mumps, let’s not forget our epidemic in 2005, a resurgence of a disease many young doctors would struggle even to recognise. Here is a graph of mumps cases from the BMJ article that analysed the outbreak:

  Almost all confirmed cases during this outbreak were in people aged fifteen to twenty-four, and only 3.3 per cent had received the full two doses of MMR vaccine. Why did it affect these people? Because of a global vaccine shortage in the early 1990s. Mumps is not a harmless disease. I’ve no desire to scare anyone—and as I said, your beliefs and decisions about vaccines are your business; I’m only interested in how you came to be so incredibly misled—but before the introduction of MMR, mumps was the commonest cause of viral meningitis, and one of the leading causes of hearing loss in children. Lumbar puncture studies show that around half of all mumps infections involve the central nervous system. Mumps orchitis is common, exquisitely painful, and occurs in 20 per cent of adult men with mumps: around half will experience testicular atrophy, normally in only one testicle, but 15 to 30 per cent of patients with mumps orchitis will have it in both testicles, and of these, 13 per cent will have reduced fertility.