by Ben Goldacre
Finally, I have no doubt that there will be some mistakes in this book, whether they are small slips, misinterpretations or perhaps unfair slurs. I should say that I’ve written it to illustrate underlying themes, not to rubbish any particular drug or company; and so I hope the criticism has been roughly evenly distributed, perhaps according to market share. I certainly don’t think any one company is any better than any other. If you do find an error of fact, do let me know, and if it’s a genuine error, I will happily correct the text. In the unlikely event that any example I’ve used is simply wrong, there will be another to slot in its place. If you like – if it’s in your nature, and if that is how you’d like others to see you – you can point out mistakes with self-righteous fury. Or you can just point them out. I’ll be cheerful in either case, but more than anything: I’m certain there is no error that will change the argument of the book, so your feedback will help to make the argument stronger.
On a related note, in the UK (especially) there is a vogue for large companies to sue writers over critical concerns they have raised, in the public interest, on matters of science and health. I’ve experienced success in libel cases, and have helped to drive a partially successful campaign to change the libel laws in Britain. Even where libel cases have been technically successful – though for clarity, none against me has ever been – these have often backfired on the litigant’s reputation. There is a strong sense among the public that libel is used to dissuade people from raising legitimate concerns, or to create anxiety, encouraging writers to police themselves and steer away from anything critical. I mention this because, as I said, I’ve tried very hard to be accurate in this book.
If you genuinely feel that you or your company has been libelled, or that something in here is plainly untrue, I encourage you to drop me a note, so we can look at your concerns and change what’s written, if appropriate, or clarify. I offer this freely, with no sense of either fear or threat: I just think it’s how things should work. As I have repeatedly reiterated throughout this book, the problems it describes are systemic and widespread. The specific stories I have included are intended to illustrate points of methodology, and those points would only make sense if they were attached to real studies. I hope you’ll view any story that relates to you in the spirit in which it is intended, and recognise the genuine concern and public interest in the issues raised, as well as the room for improvement in your industry.
NOTES
Chapter 1: Missing Data
1 Bourgeois FT, Murthy S, Mandl KD. Outcome Reporting Among Drug Trials Registered in ClinicalTrials.gov. Annals of Internal Medicine. 2010;153(3):158–66.
2 Bero L, Oostvogel F, Bacchetti P, Lee K. Factors Associated with Findings of Published Trials of Drug–Drug Comparisons: Why Some Statins Appear More Efficacious than Others. PLoS Med. 2007 Jun 5;4(6):e184.
3 Kelly RE Jr, Cohen LJ, Semple RJ, Bialer P, Lau A, Bodenheimer A, et al. Relationship between drug company funding and outcomes of clinical psychiatric research. Psychol Med. 2006 Nov;36(11):1647–56.
4 Bekelman JE, Li Y, Gross CP. Scope and impact of financial conflicts of interest in biomedical research: a systematic review. JAMA 2003;289:454–65. Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ 2003;326:1167–70.
5 Sergio S. Pharmaceutical company funding and its consequences: A qualitative systematic review. Contemporary Clinical Trials. 2008 Mar;29(2):109–13.
6 Eyding D, Lelgemann M, Grouven U, Harter M, Kromp M, Kaiser T, et al. Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ. 2010 Oct 12;341:c4737–c4737.
7 Suntharalingam G, Perry MR, Ward S, Brett SJ, Castello-Cortes A, Brunner MD, et al. Cytokine storm in a phase 1 trial of the anti-CD28 monoclonal antibody TGN1412. N. Engl. J. Med. 2006 Sep 7;355(10):1018–28.
8 Expert Group on Phase One Clinical Trials: Final report [Internet]. 2006 [cited 2012 Apr 5]. Available from: http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_063117
9 Decullier E, Chan A-W, Chapuis F. Inadequate Dissemination of Phase I Trials: A Retrospective Cohort Study. PLoS Med. 2009 Feb 17;6(2):e1000034.
10 Cowley AJ, Skene A, Stainer K, Hampton JR. The effect of lorcainide on arrhythmias and survival in patients with acute myocardial infarction: an example of publication bias. International journal of cardiology. 1993;40(2):161–6. Iain Chalmers was the first to raise TGN1412 and anti-arrhythmics as examples of the harm done when individual early trials are left unpublished. They are the best illustrations of this problem, but you should not imagine that they are unusual: the quantitative data shows that they are just two among many, many similar cases.
11 Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992 Jul 8;268(2):240–8.
12 Here is the classic early paper arguing this point: Chalmers Iain. Underreporting Research Is Scientific Misconduct. JAMA. 1990 Mar 9;263(10):1405–1408.
13 Sterling T. Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. Am Stat Assoc J 1959;54:30–4.
14 Sterling TD, Rosenbaum WL, Weinkam JJ. Publication decisions revisited – the effect of the outcome of statistical tests on the decision to publish and vice-versa. Am Stat 1995;49:108–12.
15 Bacon F (1645). Franc Baconis de Verulamio/Summi Angliae Cancellarii/Novum organum scientiarum. [Francis Bacon of St. Albans Lord Chancellor of England. A ‘New Instrument’ for the sciences] Lugd. Bat: apud Adrianum Wiingaerde, et Franciscum Moiardum. Aphorism XLVI (p.45–46).
16 Fowler T (1786). Medical reports of the effects of arsenic in the cure of agues, remitting feveres and periodic headachs. London: J Johnson, pp 105–107.
17 Hemminki E. Study of information submitted by drug companies to licensing authorities. Br Med J. 1980 Mar 22;280(6217):833–6.
18 Lee K, Bacchetti P, Sim I. Publication of clinical trials supporting successful new drug applications: a literature analysis. PLoS Med 2008;5(9):e191.
19 Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine – selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ 2003;326:1171–3.
20 Rising K, Bacchetti P, Bero L. Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation. PLoS Med. 2008 Nov 25;5(11):e217.
21 Scherer RW, Langenberg P, von Elm E. Full publication of results initially presented in abstracts. Cochrane Database Syst Rev 2007; 2: MR000005.
22 Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010 Feb;14(8):iii, ix–xi, 1–193.
23 Dickersin K. How important is publication bias? A synthesis of available data. Aids Educ Prev 1997;9(1 SA):15–21.
24 Ioannidis J. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 1998;279:281–6.
25 Bardy AH. Bias in reporting clinical trials. Brit J Clin Pharmaco 1998;46:147–50.
26 Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008;3(8):e3081.
27 Decullier E, Lhéritier V, Chapuis F. Fate of biomedical research protocols and publication bias in France: retrospective cohort study. BMJ 2005;331:19. Decullier E, Chapuis F. Impact of funding on biomedical research: a retrospective cohort study. BMC Public Health 2006;6:165.
28 Cronin E, Sheldon T. Factors influencing the publication of health research. Int J Technol Assess 2004;
20:351–5.
29 Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010 Feb;14(8):iii, ix–xi, 1–193.
30 This was first pointed out to me by Jamie Heywood from PatientsLikeMe, who spent huge resources trying and failing to replicate research findings in another area of medicine. The last time I saw him we talked about writing up his idea that the likelihood of a claim being true is proportional to the cost of making it, and inversely proportional to the cost of refuting it. We’ve not done so, and until then, a description of our conversation is the only reference for this neat idea.
31 Begley CG, Ellis LM. Drug development: Raise standards for preclinical cancer research. Nature. 2012 Mar 28;483(7391): 531–3.
32 Harrabin R et al (2003). Health In The News, The King’s Fund, London, UK.
33 Forsyth, Alasdair J. M. 2001. Distorted? a quantitative exploration of drug fatality reports in the popular press. International Journal of Drug Policy 12, no. 5–6 (November 1): 435–453.
34 Dickersin K, Min YI, Meinert CL: Factors influencing publication of research results: follow-up of applications submitted to two institutional review boards. JAMA 1992, 267:374–378.
35 Olson CM, Rennie D, Cook D, Dickersin K, Flanagin A, Hogan JW, Zhu Q, Reiling J, Pace B: Publication bias in editorial decision making. JAMA 2002, 287:2825–2828.
36 Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust 2006;184:621–6. Lynch JR, Cunningham MRA, Warme WJ, Schaad DC, Wolf FM, Leopold SS. Commercially funded and United States-based research is more likely to be published; good-quality studies with negative outcomes are not. J Bone Joint Surg Am 2007;89:1010–8. Okike K, Kocher MS, Mehlman CT, Heckman JD, Bhandari M. Publication bias in orthopaedic research: an analysis of scientific factors associated with publication in the Journal of Bone and Joint Surgery. J Bone Joint Surg Am 2008;90:595–601.
37 Epstein WM. Confirmation response bias among social work journals. Sci Techol Hum Values 1990;15:9–38.
38 Mahoney MJ. Publication prejudices: an experimental study of confirmatory bias in the peer review system. Cognitive Ther Res 1977;1:161–75.
39 Ernst E, Resch KL. Reviewer bias – a blinded experimental study. J Lab Clin Med 1994;124:178–82.
40 Abbot NE, Ernst E. Publication bias: direction of outcome less important than scientific quality. Perfusion 1998;11:182–4.
41 Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, Leopold SS. Testing for the Presence of Positive-Outcome Bias in Peer Review: A Randomized Controlled Trial. Arch Intern Med. 2010 Nov 22;170(21):1934–9.
42 Weber EJ, Callaham ML, Wears RL, Barton C, Young G. Unpublished research from a medical specialty meeting: why investigators fail to publish. JAMA 1998;280:257–9.
43 Kupfersmid J, Fiala M. A survey of attitudes and behaviors of authors who publish in psychology and education journals. Am Psychol 1991;46:249–50.
44 Song F, Parekh S, Hooper L, Loke YK, Ryder J, Sutton AJ, et al. Dissemination and publication of research findings: an updated review of related biases. Health Technol Assess. 2010 Feb;14(8):iii, ix–xi, 1–193.
45 Gøtzsche PC, Hróbjartsson A, Johansen HK, Haahr MT, Altman DG, Chan A-W: Constraints on publication rights in industry-initiated clinical trials. JAMA 2006, 295:1645–1646.
46 Gornall, J. ‘Industry attack on academics.’ BMJ 338, no. mar09 1 (March 9, 2009): b736–b736.
47 Ibid.
48 Steinbrook R. Gag clauses in clinical-trial agreements. N. Engl. J. Med. 2005 May 26;352(21):2160–2.
49 Mello MM, Clarridge BR, Studdert DM. Academic medical centers’ standards for clinical-trial agreements with industry. N. Engl. J. Med. 2005;352(21):2202.
50 This is one of many stories for which I recommend delving into the horrible details, if you’re interested. A good place to start here is Prof David Colquhoun’s blog on the topic, with many links http://www.dcscience.net/?p=193 and this BMJ piece written by a lawyer, to keep the lawyers reading this book happy: Dyer C. Aubrey Blumsohn: Academic who took on industry. BMJ. 2009 Dec 15;339(dec15 1):b5293–b5293.
51 Wendler D, Krohmal B, Emanuel EJ, Grady C, for the ESPRIT Group. Why Patients Continue to Participate in Clinical Research. Arch Intern Med. 2008 Jun 23;168(12):1294–9.
52 McDonald AM, Knight RC, Campbell MK, Entwistle VA, Grant AM, Cook JA, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials. 2006;7:9.
53 Simes RJ. Publication bias: the case for an international registry of clinical trials. Journal of Clinical Oncology. 1986 Oct 1;4(10):1529–1541.
54 Clarke M, Clarke L, Clarke T. Yes Sir, no Sir, not much difference Sir. JRSM. 2007 Dec 1;100(12):571–572.
55 Chalmers Iain. Underreporting Research Is Scientific Misconduct. JAMA: The Journal of the American Medical Association. 1990 Mar 9;263(10):1405–1408.
56 Chalmers I. From optimism to disillusion about commitment to transparency in the medico-industrial complex. JRSM. 2006 Jul 1;99(7):337–341.
57 Their delegation was led by Frank Wells: his textbook on fraud is fantastic. I tell you this because you should understand that these are not all bad people with inherently secretive natures.
58 Sykes R. Being a modern pharmaceutical company. BMJ. 1998 Oct 31;317(7167):1172–80.
59 De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, et al. Clinical trial registration: a statement from the International Committee of Medical Journal Editors. The Lancet. 2004 Sep 11;364(9438):911–2.
60 Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of Registered and Published Primary Outcomes in Randomized Controlled Trials. JAMA. 2009 Sep 2;302(9):977–84.
61 Wieseler B, McGauran N, Kaiser T. Still waiting for functional EU Clinical Trials Register. BMJ. 2011 Jun 20;342(jun20 2):d3834–d3834.
62 Prayle AP, Hurley MN, Smyth AR. Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: cross sectional study. BMJ. 2012;344:d7373.
63 A good (but brief) overview of how to try and get info from non-academic sources is here: Chan A-W. Out of sight but not out of mind: how to search for unpublished clinical trial evidence. BMJ. 2012 Jan 3;344(jan03 2):d8013–d8013.
64 You can read the letters and the report online. It’s a gripping read, with many interesting and nefarious details, so I highly recommend doing so: Medicines and Healthcare products Regulatory Agency (MHRA) www.mhra.gov.u. GSK investigation concludes [Internet]. [cited 2012 Apr 29]. Available from: http://www.mhra.gov.uk/Howweregulate/Medicines/Medicinesregulatorynews/CON014153
65 This was SmithKline Beecham, before they merged with GlaxoWellcome and became GSK.
66 Lenzer J, Brownlee S. Antidepressants: an untold story? BMJ 2008;336:532–4.
67 Wood AJ. Progress and deficiencies in the registration of clinical trials. N Engl J Med. 2009;360(8):824–830
68 O’Connor AB. The need for improved access to FDA reviews. JAMA: The Journal of the American Medical Association. 2009;302(2):191.
69 http://www.prescrire.org/editoriaux/EDI33693.pdf
70 Decision of the European Ombudsman closing his inquiry into complaint 2560/2007/BEH against the European Medicines Agency. November 2010. http://www.ombudsman.europa.eu/cases/decision.faces/en/5459/html.bookmark.
71 UK drug regulator destroys all the evidence after 15 years/BMI[Internet]. Available from http://www.bmj.com/rapid-response/2011/11/03/uk-drug-regulator-destroys-all-evidence-after-15-years.
72 You might be unsurprised to hear that no large drug company has ever been prosecuted under the safety monitoring regulations in the UK.
73 This story is spread over various publications by the Cochrane team, and the account here is taken from their work, published responses from Roche, and discussions with the Cochrane tea
m. The best place to get the early half of this story is this paper: Doshi P. Neuraminidase inhibitors – the story behind the Cochrane review. BMJ. 2009;339. And for the second half, I recommend this open-access paper: Doshi P, Jefferson T, Del Mar C (2012) The Imperative to Share Clinical Study Reports: Recommendations from the Tamiflu Experience. PLoS Med 9(4): e1001201. doi:10.1371/journal.pmed.1001201 http://bit.ly/HIbwqO
74 This is a fascinating and messy new area. The paper below gives a good summary of the importance of analysing full trial programmes, and the discrepancies found on Tamiflu between papers and Clinical Study Reports: Jefferson T, Doshi P, Thompson M, Heneghan C, Group CARI. Ensuring safe and effective drugs: who can do what it takes? BMJ. 2011 Jan 11;342(jan11 1):c7258–c7258.
75 This is all from: Jefferson T, Doshi P, Thompson M, Heneghan C, Group CARI. Ensuring safe and effective drugs: who can do what it takes? BMJ. 2011 Jan 11;342(jan11 1):c7258–c7258.
76 Tom Jefferson, Lecture on Tamiflu, BMJ Evidence 2011, London.
77 Tramèr MR, Reynolds DJ, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis: a case study. BMJ. 1997 Sep 13;315(7109):635–40.
78 Doshi P, Jefferson T, Del Mar C (2012) The Imperative to Share Clinical Study Reports: Recommendations from the Tamiflu Experience. PLoS Med 9(4): e1001201. doi:10.1371/journal. pmed.1001201 http://bit.ly/HIbwqO
79 Cohen D (2009) Complications: tracking down the data on oseltamivir. BMJ 339: b5387.
80 If you’re interested in this story, the links to primary documents are all here: Diabetes drug ‘victory’ is really an ugly story about incompetence. Ben Goldacre, The Guardian. 2010 Jul 17 [cited 2012 May 2]; Available from: http://www.badscience.net/2010/07/pharmaco-epidemiology-would-be-fascinating-enough-even-if-society-didnt-manage-it-really-really-badly/
81 Nissen SE. Setting the record straight. JAMA. 2010 Mar 24; 303(12):1194–5
82 Eichler H-G, Abadie E, Breckenridge A, Leufkens H, Rasi G. Open Clinical Trial Data for All? A View from Regulators. PLoS Med. 2012 Apr 10;9(4):e1001202.