Bad Science
Page 8
Sometimes NICE is able to access some extra unpublished data from the drug companies: this is information that doctors and patients aren’t allowed to see, despite the fact that they are the people making decisions about whether to prescribe the drugs, or are actually taking them. But when NICE does get information in this way, it often comes with strict conditions on confidentiality, leading to some very bizarre documents being published. On the next page, for example, is the NICE document discussing whether it’s a good idea to have Lucentis, an extremely expensive drug, costing well over £1,000 per treatment, that is injected into the eye for a condition called acute macular degeneration.
As you can see, the NICE document on whether this treatment is a good idea is censored. Not only is the data on the effectiveness of the treatment blanked out by thick black rectangles, in case any doctor or patient should see it, but absurdly, even the names of some trials are missing, preventing the reader from even knowing of their existence, or cross referencing information about them. Most disturbing of all, as you can see in the last bullet point, the data on adverse events is also censored. This is a perfectly common state of affairs, and I’m reproducing the whole page here because I worry that it would otherwise be almost too bizarre for you to believe.
Why shouldn’t we all – doctors, patients and NICE – have free access to all this information? This is something I asked both Kent Woods from the MHRA, and Hans Georg Eichler, Medical Director of the European Medicines Agency, in 2010. Both, separately, gave me the same answer: people outside the agencies cannot be trusted with this information, because they might misinterpret it, either deliberately or through incompetence. Both, separately – though I guess they must chat at parties – raised the MMR vaccine scare, as the classic example of how the media can contrive a national panic on the basis of no good evidence, creating dangerous public-health problems along the way. What if they released raw safety data, and people who don’t know how to analyse it properly found imaginary patterns, and created scares that put patients off taking life-saving medication?
I accept that this is a risk, but I also believe their priorities are wrong: I think that the advantages of many eyes working on these vitally important problems are enormous, and the possibility of a few irrational scaremongers is no excuse for hiding data. Drug companies and regulators also both say that you can already get all the information you need from regulators’ websites, in summary form.
We shall now see that this is untrue.
Two: Regulators make it hard to
access the data they do have
When exposed to criticism, drug companies often become indignant, and declare that they already share enough data for doctors and patients to be informed. ‘We give everything to the regulator,’ they say, ‘and you can get it from them.’ Similarly, regulators insist that all you need to do is look on their website, and you will easily find all the data you need. In reality, there is a messy game, in which doctors and academics trying to find all the data on a drug are sent around the houses, scrabbling for information that is both hard to find and fatally flawed.
Firstly, as we’ve already seen, regulators don’t have all the trials, and they don’t share all the ones that they do. Summary documents are available on the early trials used to get a drug onto the market in the first place, but only for the specific licensed uses of the drug. Even where the regulator has been given safety data for off-label uses (following the paroxetine case above) the information from these trials still isn’t made publicly available through the regulator: it simply sits quietly in the regulator’s files.
For example: duloxetine is another SSRI drug in fairly widespread use, which is usually given as an antidepressant. During a trial on its use for a completely different purpose – treating incontinence – there were apparently several suicides.66 This is important and interesting information, and the FDA hold the relevant data: it conducted a review on this issue, and came to a view on whether the risk was significant. But you cannot see any of that on the FDA website, because duloxetine never got a licence for use in treating incontinence.67 The trial data was only used by the FDA to inform its internal ruminations. This is an everyday situation.
But even when you are allowed to see trial results held by regulators, getting this information from their public websites is supremely tricky. The search functions on the FDA website are essentially broken, while the content is haphazard and badly organised, with lots missing, and too little information to enable you to work out if a trial was prone to bias by design. Once again – partly, here, through casual thoughtlessness and incompetence – it is impossible to get access to the basic information that we need. Drug companies and regulators deny this: they say that if you search their websites, everything is there. So let’s walk, briefly, through the process in all its infuriating glory. The case I will use was published three years ago in JAMA as a useful illustration of how broken the FDA site has become:68 replicating it today, in 2012, nothing has changed.
So: let’s say we want to find the results from all the trials the FDA has, on a drug called pregabalin, in which the drug is used to treat pain for diabetics whose nerves have been affected by their disease (a condition called ‘diabetic peripheral neuropathy’). You want the FDA review on this specific use, which is the PDF document containing all the trials in one big bundle. But if you search for ‘pregabalin review’, say, on the FDA website, you get over a hundred documents: none of them is clearly named, and not one of them is the FDA review document on pregabalin. If you type in the FDA application number – the unique identifier for the FDA document you’re looking for – the FDA website comes up with nothing at all.
If you’re lucky, or wise, you’ll get dropped at the Drugs@FDA page: typing ‘pregabalin’ there brings up three ‘FDA applications’. Why three? Because there are three different documents, each on a different condition that pregabalin can be used to treat. The FDA site doesn’t tell you which condition each of these three documents is for, so you have to use trial and error to try to find out. That’s not as easy as it sounds. I have the correct document for pregabalin and diabetic peripheral neuropathy right here in front of me: it’s almost four hundred pages long, but it doesn’t tell you that it’s about diabetic peripheral neuropathy until you get to page 19. There’s no executive summary at the beginning – in fact, there’s no title page, no contents page, no hint of what the document is even about, and it skips randomly from one sub-document to another, all scanned and bundled up in the same gigantic file.
If you’re a nerd, you might think: these files are electronic; they’re PDFs, a type of file specifically designed to make sharing electronic documents convenient. Any nerd will know that if you want to find something in an electronic document, it’s easy: you just use the ‘find’ command: type in, say, ‘peripheral neuropathy’, and your computer will find the phrase straight off. But no: unlike almost any other serious government document in the world, the PDFs from the FDA are a series of photographs of pages of text, rather than the text itself. This means you cannot search for a phrase. Instead, you have to go through it, searching for that phrase, laboriously, by eye.
I could go on. I will. There’s some kind of ‘table of contents’ on the seventeenth page, but it gets the page numbers wrong. I’ve finished now. There is simply no reason for this obfuscation and chaos. These problems aren’t caused by technical issues specific to trials, and they would hardly cost any money at all to fix. This is plainly, simply, unhelpful, and the best we can hope is that it’s driven by thoughtlessness.
That’s a tragedy, because if you can unearth this document, and decode it, you will find that it is full of terrifying gems: perfect examples of situations in which a drug company has used dodgy statistical methods to design and analyse a study, in such a way that it is predestined – from the outset – to exaggerate the benefits of the drug.
For example, in the five trials on pregabalin and pain, lots of people dropped out during the s
tudy period. This is common in medical trials, as you will shortly see, and it often happens because people have found a drug to be unhelpful, or have had bad side effects. During these trials you’re measuring pain at regular intervals. But if some people drop out, you’re left with an important question: what kind of pain score should you use for them in your results? We know, after all, that people dropping out are more likely to have done badly on the drug.
Pfizer decided to use a method called ‘Last Observation Carried Forward’, which means what you’d expect: you take the last measurement of pain severity while the patients were on the drug, from just before they dropped out, and then paste that in for all the remaining pain measures that they missed, after they stopped coming to follow-up appointments.
The FDA disapproved of this: it pointed out, quite correctly, that Pfizer’s strategy would make the drug look better than it really is. For a fairer picture, we have to assume that the dropouts stopped taking the drug because of side effects, so their pain score should reflect the reality, which is that they would never get any benefit from the drug in normal use. The correct level of pain to record for them is, therefore, their pain at the beginning of the study, before they had any kind of treatment (if you’re interested, this is called ‘Baseline Observation Carried Forward’). The analysis was duly redone, properly, and a more modest, more accurate view of the benefits of the drug was produced. In this case, it turns out that using the ‘last observation’ method overestimated the improvement in pain by about a quarter.
Here’s the catch. Four out of five of these trials were then published in the peer-reviewed academic literature, the place where doctors look for evidence on whether a drug works or not (one trial wasn’t published at all). Every single one of the published analyses used ‘Last Observation Carried Forward’, the dodgy method, the one that exaggerates the benefits of the drug. Not one of them acknowledges that ‘last observation’ is a technique that overstates these benefits.
You can see why it is important that we have access to all the information we can possibly get on every drug trial: not only are some whole trials withheld from us, but there are often hidden flaws in the methods used. The devil is in the detail, and there are many dodgy trials, as we shall soon see, with flaws that may not be clear even in the academic papers, let alone in the thin and uninformative summaries from regulators. Furthermore, as we shall also see very shortly, there are often worrying discrepancies between the regulators’ summary documents and what actually happened in the trial.
This is why we need to get hold of a more detailed document on each trial: something called the Clinical Study Report (CSR). These are long pieces of work, sometimes thousands of pages, but they are complete enough for the reader to reconstruct exactly what happened to all the participants; and they will let you find out where the bodies are buried. Drug companies give this study report to the regulator – though still only for formally licensed uses of the drug – so both have a copy, and both should be happy to hand it over.
We will now see what happens when you ask them.
Three: Regulators withhold study
reports that they do have
In 2007, researchers from the Nordic Cochrane Centre were working on a systematic review for two widely used diet drugs, orlistat and rimonabant. A systematic review, as you know, is the gold-standard summary of the evidence on whether a treatment is effective. These are life-saving, because they give us the best possible understanding of the true effects of a treatment, including its side effects. But doing this requires access to all of the evidence: if some is missing, especially if unflattering data is deliberately harder to obtain, we will be left with a distorted picture.
The researchers knew that the trial data they were able to find in the published academic literature was probably incomplete, because negative trials are routinely left unpublished. But they also knew that the European Medicines Agency (EMA) would have much of this information, since the manufacturers of drugs are obliged to give the study reports to the regulator when trying to get them onto the market. Since regulators are supposed to act in the interests of patients, they applied to the EMA for the protocols and the study reports. That was in June 2007.
In August, the EMA responded: it had decided not to give out the study reports for these trials, and was invoking the section of its rules which allows it to protect the commercial interests and intellectual property of drug companies. The researchers replied immediately, almost by return of post: there is nothing in the study reports that will undermine the protection of someone’s commercial interests, they explained. But if there was, could the EMA please explain why it felt the commercial interests of the drug companies should override the welfare of patients?
Here we should pause for just one moment, and think about what the EMA is doing. It is the regulator that approves and monitors drugs for the whole of Europe, with the aim of protecting the public. Doctors and patients can only make meaningful decisions about treatments if they have access to all the data. The EMA has that data, but has decided that the drug companies’ interests are more important. Having spoken to a lot of people in regulation, I can offer one small insight into what on earth they might be thinking. Regulators, in my experience, are preoccupied with the idea that they see all the data, and use it to make the decision about whether a drug should go on the market, and that this is enough: doctors and patients don’t need to see the data, because the regulator has done all that work.
This misunderstands a crucial difference between the decisions made by regulators and the decisions made by doctors. Contrary to what some regulators seem to think, a drug is not either ‘good’ and therefore on the market, or ‘bad’ and therefore off it. A regulator makes a decision about whether it’s in the interests of the population as a whole that the drug should be available for use, at all, ever – even if only in some very obscure circumstance, infrequently and cautiously. This bar is set pretty low, as we shall see, and lots of drugs that are on the market (in fact, the overwhelming majority) are hardly ever used.
A doctor needs to use the same information as that available to the regulator in order to make a very different decision: is this the right drug for the patient in front of me right now? The simple fact that a drug is approved for prescription doesn’t mean it’s particularly good, or the best. In fact, there are complex decisions to be made in each clinical situation about which drug is best. Maybe the patient has failed to get better on one drug, so you want to try another, from a different class of drugs; maybe the patient has mild kidney failure, so you don’t want to use the most popular drug, as that causes very occasional problems in patients with dodgy kidneys; maybe you need a drug that won’t interfere with other drugs the patient is taking.
These complex considerations are the reason we are OK with having a range of drugs on the market: even if some of them are less useful overall, they might be useful in specific circumstances. But we need to be able to see all of the information about them, in order to make these decisions. It is not enough for the regulators to grandly state that they have approved a drug, and therefore we should all feel happy to prescribe it. Doctors and patients need the data just as much as regulators do.
In September 2007 the EMA confirmed to the Cochrane researchers that it wasn’t going to share the study reports on orlistat and rimonabant, and explained that it had a policy of never disclosing the data given as part of a marketing authorisation. A serious problem had emerged. These weight-loss drugs were being widely prescribed throughout Europe, but doctors and patients were unable to access important information about whether they worked, how bad the side effects were, which was more effective, or any of a whole host of other important questions. Real patients were being exposed to potential harm, in everyday prescribing decisions, through this lack of information that was enforced by the EMA.
The researchers went to the European Ombudsman with two clear allegations. Firstly, the EMA had failed to give sufficient reaso
ns for refusing them access to the data; and secondly, the EMA’s brief, dismissive claim that commercial interests must be protected was unjustified, because there was no material of commercial interest in the trial results, other than the data on safety and effectiveness, which doctors and patients obviously need to access. They didn’t know it at the time, but this was the beginning of a battle for data that would shame the EMA, and would last more than three years.
It took four months for the EMA to respond, and over the next year it simply reiterated its position: as far as it was concerned, any kind of information the disclosure of which would ‘unreasonably undermine or prejudice the commercial interests of individuals or companies’ was commercially confidential. The study reports, it said, might contain information about the commercial plans for the drug. The researchers responded that this was unlikely, but was in any case of marginal importance, as part of a much more important and pressing situation: ‘As a likely consequence of [the] EMA’s position, patients would die unnecessarily, and would be treated with inferior and potentially harmful drugs.’ They regarded the EMA’s position as ethically indefensible. More than that, they said, the EMA had a clear conflict of interest: this data could be used to challenge its summary views on the benefits and risks of these treatments. The EMA had failed to explain why doctors and patients having access to study reports and protocols should undermine anyone’s reasonable commercial interests, and why these commercial interests were more important than the welfare of patients.