Not only are negative results often unavailable, but the influence of positive results is also often exaggerated. Studies with positive results are: more likely to be published quickly; more likely to generate multiple publications; more likely to be published in a “high impact” journal; more likely to be cited by other investigators; and more likely to be published in English. Those are all reasons why experts especially value positive results when they sit down to weigh the evidence. So the experts’ evidence, a distillation of the available information, could be distorted by unknown negative results and by over-valued positive results.
Does this actually happen at the real-world interface of the known and unknown amid the babble and clatter of scientific discovery? The answer is probably yes, but you don’t have to take our word for it.
“Published results of scientific investigations are not a representative sample of results of all scientific studies,” write Simon Fraser University School of Computing Science investigators T. D. Sterling, W. L. Rosenbaum, and J. J. Weinkam in a 1995 article in The American Statistician.125 They found biases toward publishing studies that showed statistically significant positive results in eleven major journals. This has two bad effects. One is that investigators, having no way to know about unpublished data, keep doing experiments that others, maybe several others, have proven don’t work. That wastes money and even more valuable resources—the time, energy and enthusiasm of medical investigators.
The second, follow-on effect hinges on the words statistically significant. That requirement means that smaller studies and those with less significant results—suppose your p value was 0.06—are unlikely to see the light of day. This in spite of the fact that everybody knows that p<0.05 is a completely arbitrary threshold for significance and that results with higher p values may be just as true as those that happen to make the passing statistical grade. So some things that might actually work go unreported, which means lost opportunities.
So how much of the information from clinical trials gets published and so is widely available? Probably around half. That figure is astounding, but it may be true. We’ll explain why we say it may be true below, but think for a minute about what this means. Your doctor knows that the results of maybe half of the trials of treatments that she is contemplating recommending to you may have gone unreported in the literature. Such studies would have had minimal influence on the expert panels that developed guidelines, or on the Cochrane Collaboration’s reviewers, or on the meta-
analysts. Faced with a specific person with a specific problem, how does the good doctor go about making a decision? First of all, she carefully assesses the available evidence in the context of what she knows about you. After all, even imperfect evidence is the factual basis for making health decisions, and the good doctor values facts. She also knows that there may be an unknown number of unreported studies that were either negative or inconclusive and that the reported positive studies may be over-valued by the experts who decide on the evidence. Given all of that, how does your doctor figure out what is most likely to help you get or stay well? Sooner or later you and she will have to deal with uncertainty.
What about the claim that half of the clinical trials data is never published? How can we know for sure since we are dealing with (if we have our Rumsfeldisms straight) an unknown known, that is, things that are known by someone but not by us? Well, in the case of clinical trials, one can get a pretty good idea because of the foresight of our federal government in establishing by law in 2008 the database known as ClinicalTrials.gov.
The U.S. Food and Drug Administration Act obligates clinical investigators to post the results of all clinical trials of Food and Drug Administration-approved drugs on ClinicalTrials.gov within a year of completing the study.126 So, on March 27, 2012, Parisian clinical epidemiologist Carolina Riveros and her colleagues searched that database for the results of randomized controlled trials.127 For a random sample of six hundred of those trials, they also searched PubMed, a separate database that lists essentially all medical publications. You’ve already guessed where the 50 percent number comes from. Of the six hundred trials reported in ClinicalTrials.gov, only 297 were reported in a published article. And it gets worse. In 202 studies for which there were published articles, adverse and serious adverse events were significantly under reported in the published articles compared to the government database; apparently the legal obligation outweighs the moral one . . . disappointing.
So, the available medical information may be biased toward positive results and against negative findings. Apart from complicating a doctor’s efforts to help her patients decide what is right for them, does it really matter? Yes, it matters. It can matter a lot!
While we have no inside information about what drives British medical researcher Andrew Wakefield or about the review process of the prestigious British medical journal The Lancet, we, and the rest of the world, know now that the 1998 Lancet paper claiming that childhood vaccinations cause autism was bogus.128 The editors of The Lancet know that too; they retracted the article in 2010, rather late in the wake of a deluge of scientific, professional and social criticism of the motives, performance, and reporting of Wakefield’s studies. If it wasn’t a less than healthy fascination with dramatic positive results, then we have no idea why The Lancet accepted this article, but we would bet the farm that, before that article appeared, an identical study concluding that there was no link between vaccination and autism would have been soundly rejected!
We earlier referred to Wakefield et al.’s Lancet paper as “the holy text” of the anti-vaccine movement, a movement with real consequences. More susceptible children sets the stage for the reappearance of diseases that vaccination programs had largely eliminated from this country. The multistate measles epidemic in 2015 is a dramatic illustration.129 Medical researchers’ and their journals’ relentless pursuit of positive and interesting results sometimes has serious consequences that are not good, and once the erroneous information genie is out there it is hard to force the wily rascal back into the bottle.
Recognizing the potential and real downside of publication bias in medicine, scientists have given some serious thought to possible remedies. Here are some possibilities that have been considered: link research funding for clinical studies to quantity of publications (since most researchers have more negative than positive data, this might get some more of the negative stuff into circulation); continue efforts to create a culture among investigators, academic institutions, and elite journals that values well done and definitive negative studies (culture change is hard, good luck); create negative results sections in medical journals; and create new journals specifically for publishing negative results.
David Alcantara Parra, Spanish chemist and nanoscientist, created the All Results Journals (ARJournals) specifically intended to “make use of all the data that go unreported within the scientific community.”130 Building on the premise that “failure is as important in science as in other aspects of life,” the Journal of Negative Results in Biomedicine (JNRBM) was launched in 2002 with Harvard Professor of Developmental Biology Bjorn Olsen as its editor.131 The focus of the journal is in ecology and evolutionary biology, but there could be value in emulating it in fields more directly relevant to the practice of medicine.
The good doctor knows all of this. She knows how important the published literature is and studies it religiously, but she also knows that like most things that humans do, the literature is not perfect. So how does she go about getting as close as possible to a reasonably good look at a complete and accurate picture?
Were you to barge into the office unannounced, you might well find the good doctor staring intently at a computer screen. She is what we will call a discriminating web surfer (to be clearly distinguished from the more numerous naïve, recreational, or gullible varieties of the species). This doctor uses all available tools (there are many, e.g. WebMD, Up-To-Date, even when desperate maybe
Google) to flatten her medical world. She values published information but does not hesitate to cast a wide net in her efforts to understand the whole picture; in the computer age, absent barriers of geography or time, she can cast an enormously wide net. Of course she realizes that a wide net collects a lot of garbage (she is no fan of bizarre conspiracy theories, and stories of alien forces or political intrigue don’t excite her), which is where her powers of discrimination become indispensable. That is why you need her help.
A lot of the negative results and some of the less statistically significant but positive results that didn’t make it into the mainstream medical literature are retrievable from the cloud, and those results may fill in some details of the picture outlined in the published literature. That is, if one can, like the famous detective, “recognize, out of a number of facts, which are incidental and which are vital.”132
A thinking doctor knows that contributions to the internet are often not critically curated, so she starts by sorting out the needles of facts from the haystack of fictions. If you try to do this all by yourself at home, you are sure to make mistakes; we are all more vulnerable to the influence of misinformation than we would like to admit. This good doctor spent a lot of years learning the stuff one needs to know to pick out what is true and important. That’s one reason why insurance companies pay doctors. No thoughtful doctor gets her legal advice from bartenders, relatives, or anyone else who posts their ruminations on the internet; she consults a lawyer. You are wise to deal with your health care needs similarly; consult a doctor.
Only a fraction of what fills the widely cast net of even the most dedicated and discriminating web surfer will be actual data. There are a lot of opinions, musings, theories (too often ill-founded and almost always untested), confabulations, inane comments on this or that, the stuff of the haystack. But this doctor is looking for data.
“Data! Data! Data!” Sherlock Holmes cried, “I can’t make bricks without clay.”133
In medicine, data with a capital D is on the way. It is said and published in multiple places that a sea change is afoot that will not just rewrite the rules, but will make health care an entirely new game. Will the imminent age of big data run contemplative doctors out of business? Probably not, but their job and how they do it will change a lot.
There will surely be seismic changes in how fast information gets out and what can be done with it. Digital visionaries writing in The Fourth Paradigm imagine a time of health care singularity, meaning real time generation, processing, analysis, communication, and integration of data related to human health.134 Enormous amounts of data, assembled into functional networks and available as interactive models, could give the future doctor an entirely new set of tools for accessing and interpreting research results. Journals as we know them could become relics of historical interest but of little practical use.
And the age of big data for health care is imminent, ushered in by the deluge of medical information being collected and stored, coupled with quantum advances in the technology that promise to convert those data into understanding. There are a lot of things still to be worked out, but the potential boggles the mind. Might the time come when you and your doctor can get to all of your health-related data instantly, no matter where it was collected, in a single place in the cloud? That would be a major advance. But then if your data were put in the context of the relevant data from all of your fellow humans and all of the available research on the subject you’re worried about, analyzed, and the implications spelled out for you, your doctor might start to get a little nervous about your need for her. You would have not just the clay but the bricks already made.
Well, you may have the bricks, but you still haven’t built the house and you can’t do that alone, even with the help of your trusty computer. The digitization of medicine, when done, will dramatically increase the availability, accuracy, and efficiency of health care, but it won’t eliminate the need for flesh-and-blood professionals. Roles will change. Doctors will need some new skills and will need to know some stuff that isn’t critical in the present health care world. But no matter how perfectly the digital world works, we human beings will still be analog; it is the nature of the beast. We will always need a wise, knowledgeable, and caring partner to help us navigate the process, and our need for the healing touch of a human hand will not go away.
“Don’t Believe Everything You Read on the Internet Just Because There’s a Picture with a Quote Next to It,” http://weknowmemes.com/wp-content/uploads/2012/07/dont-believe-everything-you-see-on-the-internet.jpg.
Colin Bennett, Politics of the Imagination: The Life, Work and Ideas of Charles Fort (New York: Cosimo Books, 2010).
Ivan Oransky, “Weekend Reads: Gay Canvassing Study Saga Continues: Elsiever Policy Sparks Concern: A String of Scandals,” Retraction Watch: Tracking Retractions as a Window into the Scientific Process, http://retractionwatch.com.
Adam Marcus and Ivan Oransky, “What’s behind Big Science Frauds?,” The New York Times, May 22, 2015.
T. D. Sterling, W. L. Rosenbaum, and J. T. Weinkam, “Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa,” The American Statistician 49 (1995): 108-112.
Nicola Jones, “Half of U.S. Clinical Trials Go Unpublished: Results Are Reported More Thoroughly in Government Data Base Than in Journals,” Nature News and Comment, December 13, 2013.
Carolina Riveros, Agnes Dechartres, Elodie Perrodeau, Romana Haneef, Isabelle Boutron, and Phillips Ravaud, “Timing and Completeness of Trial Results Posted at Clinicaltrials.gov and Published in Journals,” PLOS Medicine, doi: 10:1371/journal.pmed 1001586.
Andrew Wakefield,” Wikipedia, http://en.wikipedia.org/wiki/Andrew_Wakefield.
Terrance McCoy, “The Disneyland Measles Outbreak and the Disgraced Doctor Who Whipped up Vaccination Fear,” The Washington Post Morning Mix, January 23, 2015.
Negative Results Journals: A New Scientific Journal Publishes Negative Results,” Science News, http://topsciencenews.blogspot.com/2010/11/negative-results-journals.html.
Gabriella Anderson, Haiko Sprot, and Bjrn R. Olsen, “Opinion: Publish Negative Results: Non-confirmatory or ‘Negative’ Results Are Not Worthless,” The Scientist News and Opinion, January 15, 2013.
“Sherlock Holmes’ Top 10 Lessons for Problem Solvers,” Young Associates Inc., http://www.youngassocinc.com/problemswesolve.html.
“Sherlock Holmes’ Top 10 Lessons for Problem Solvers,” Young Associates Inc., http://www.youngassocinc.com/problemswesolve.html.
Tony Hey, Stewart Tansley, and Kristin Tolle, eds., The Fourth Paradigm: Data-Intensive Scientific Discovery (Redmond: Microsoft Research, 2000).
PART IV
Some Things to Expect from Your Doctor
CHAPTER 12
Your Story Is Front and Center—Narrative-Based Medicine
Scientific thinking might be defined as learning to distinguish the
exception from the rule. I’d have a hard time entrusting my health to someone who didn’t know the difference.
—Stan Polanski135
Ms. Sydney Blasingame (not her real name) was ninety-eight years old and still preferred to be addressed as Miss Syd.136 Her primary school students had called her that for many years and she treasured the memories of those years. Miss Syd was doing pretty well for her age, but she did have a little high blood pressure and her heart wasn’t pumping quite as well as it did in her more vigorous past. For many years, Miss Syd had been cared for by a senior physician in the clinic of a large university medical center. Her doctor, a professor in the medical school, decided to take a well-earned sabbatical leave to study some new laboratory techniques and during the time he was away, Miss Syd’s care was taken over by one of the
professor’s younger colleagues. When the young doctor reviewed her chart, he was appalled to learn that Miss Syd was being treated with medicines that were many decades out of date. Didn’t the professor know that there were much better medicines for treating her blood pressure and heart failure? So the young doctor would need to radically change this woman’s medical regimen.
“Ms. Blasingame,” the doctor greeted his new patient on their first meeting.
“Miss Syd,” she corrected him, not too pleased with this substitute doctor who looked young enough to be her great-grandchild.
“Miss Syd,” the doctor responded. “Tell me about yourself.”
There ensued a long conversation with Miss Syd doing most of the talking and the doctor most of the listening. What the doctor learned was that Miss Syd was still living alone and caring for herself but had some trouble keeping things straight in her mind. She had learned to compensate for that by carefully organizing the things that were really important for her to do. For example, she knew exactly what each of the pills she was taking looked like and associated their appearance with the prescribed dose and frequency—the little white pill was once each morning, the larger oval pill was once in the morning and once at bedtime. She had this regimen down pat and it hadn’t been changed in several years.
The doctor pondered Miss Syd’s story. She had lived almost a century, was doing pretty well, and making certain that she took medicines right required a lot of effort on her part. Changing to the new pills he was considering would not be so easy for her. They looked different than the pills she had been taking and the timing of the doses would be different. The new regimen would be a big change in Miss Syd’s life and it wasn’t certain, even if she managed to take the pills as directed, that she would feel a lot better than she felt now. The doctor didn’t change Miss Syd’s medicines and she continued to do well. The rules would have said to change her to the pills that had been shown in large controlled clinical trials to be more effective than the ones she was taking. But Miss Syd was an exception. She had her own special story.
The Good Doctor Page 12