Snowball in a Blizzard

by Steven Hatch

  It is also at this edge of resolution that we have to consider that people sometimes are different—in their genetics, their lifestyles, and their surroundings—and we may be pulling marbles out of the jar and making conclusions based on statistical significance that cannot be generalized to all the marbles on the planet. If we were to perform statin studies on fifty-year-olds in Botswana, we might miss the value the drug has for people living in industrialized nations because the mortality rates from HIV, tuberculosis, and malaria are so high that any lifesaving effect would be negated by premature death from something else—in other words, there are too many red marbles in the jars for this kind of a drug trial. Likewise, if we were to perform statin experiments on a similar cohort in some Scandinavian country where everyone lives to 115 and a 50-year-old is thought of as something like an adolescent, there are too many blue marbles in the jars of this experiment to see any effects. As I illustrated in the arguments about blood pressure trials, the so-called FEVER study (cited by advocates of a lower target blood pressure) was performed in China. There are many reasons to suspect that this study can’t be generalized to Americans or Europeans or anyone else. Alternately, two studies from Japan that found the opposite (no clear benefits from lowering blood pressure past a certain point) should also be considered with the same level of caution.

  Other small effects may come in medications or lifestyles or procedures that don’t save lives but merely improve the quality of life. You can find many people who will swear to you about the benefits of any number of treatments for the common cold. I’m often asked in professional and personal settings about zinc and whether that “helps.” The simplest answer is, nobody knows, and that’s due to a multiplicity of reasons. To really find out, in a scientifically meaningful way, that zinc can shorten the duration of the common cold, you would have to do a trial that satisfied many criteria. First, you would have to make sure that people who have symptoms that resemble the common cold actually have a viral infection and not some other problem like seasonal allergies; otherwise, you wouldn’t know whether the fact that zinc failed to show benefit was because it was being given to the wrong population. Then you would have to enroll most patients at about the same time course of their illness because if zinc helps at all, it helps only in shortening the time people are sick, as nobody dies from the common cold. Finally you would have to figure out how you’re going to actually measure improvement with enough precision that you could be confident that your results really do show a benefit to zinc. Measuring mortality, for instance, is easy: there is generally very little confusion among researchers when a study subject dies. But trying to find a scale for sniffles or blechiness—the kind of symptoms seen with the common cold—is much harder to design and quantify. They are “fuzzy variables,” difficult to reproduce, quite subjective, and easy to manipulate, even if unintentionally.

  To perform such a study with scientific exactitude, therefore, would be very expensive and probably require thousands of people. Ultimately, it would need to be funded by a group for whom cost either was not the principal concern (i.e., government) or was the overriding concern (a drug company). Because zinc cannot be patent protected, no multibillion-dollar pharmaceutical company will ever be interested in taking on a clinical zinc trial. Likewise, the NIH mercifully does not consider this a research priority, so there the matter rests, assuming nobody at the National Center for Complementary and Alternative Medicine finds the question absorbing enough to allocate funds to study it.

  Do I recommend against people taking zinc for the common cold? No, not really. Do I take zinc myself when I am similarly afflicted? No, not really, unless it will mollify some well-meaning friend who feels impelled to fob said panacea upon me. Zinc for the cold is simply a question that has no answer, and, as an over-the-counter supplement, it’s basically harmless—though more on that with respect to similar treatments in a moment.

  Virtually all of the elixirs found in “healthy living” aisles and nutrition stores and the like resemble zinc in that if they have any beneficial effects at all, they are sufficiently small that they either can’t be studied for reasons similar to what we’ve witnessed with zinc, or they won’t be studied because there is no profit in it for potentially interested manufacturers. There are the occasional exceptions: Saint John’s wort has been studied in the treatment of mild depression, and although the evidence supporting its effectiveness is mixed at best, at least it’s been studied. Other over-the-counter herbals that have been evaluated include saw palmetto (prostatic hypertrophy), echinacea (common cold), and ginkgo biloba (memory loss). For the most part, no benefits have been found for any of these herbals, but given the strong likelihood that whatever beneficial effects they have are not only weak but very weak, well-designed trials on these drugs are largely wanting.

  The majority of herbals, however, haven’t been studied at all—and the special concoctions made by some companies as they vie for customer loyalty have combinations and home-cooked ingredients that are nearly impossible to study in any standardized way. Mostly, they’re harmless except to the pocketbook, although some of them can have either interactions with prescription drugs or come with their own dangers. In the early 2000s, for instance, a rash of deaths from the use of an herbal used for weight loss known as ma huang, or ephedra, led the Food and Drug Administration to ban its sale in the United States. That would be not so benign.

  Weak effects also make it hard to know what really constitutes the healthiest diet, another topic that is covered ad nauseam, as it were, by health reporters. Based on many observational studies, it’s very clear that the combination of modest eating and frequent exercise has lifesaving benefits. What kind of food is associated with the greatest health is a much more difficult question to assess, although it never stops some commercially driven personality from hawking the latest fad diet or food. While this book has been in preparation, the açaí berry has become ever more revered by a particular slice of the American public bent on maximizing its health. The rapid rise in popularity of this fruit, which grows mainly in Brazil, is causing hardships for locals who had eaten it for subsistence but now cannot due to the commercial demands for this product, which is usually consumed by people who are in turn likely to donate to causes like the loss of traditional living in Amazonian rain forest that their own consumer habits are driving.*

  See, for instance, Adriana Brasileiro’s article, “Superfood Promoted on Oprah’s Site Robs Amazon Poor of Staple,” on the Bloomberg website, May 14, 2009, http://www.bloomberg.com/apps/news?pid=newsarchive&sid=ai8WCgSJrhmY&refer=environment. Dr. Mehmet Oz, in touting the berry during a show in 2008, noted, “It has twice the antioxidant content of a blueberry.” The follow-up question should have been, So what? There is just so much pseudoscientific nonsense in this one assertion that it’s hard to know where to begin, and although it’s not a big deal for people to favor açaí to blueberry in the almost certainly mistaken belief that it is somehow healthier because it has twice the antioxidant content, it has become a very big deal for poor families living in the Amazon who have been robbed of a staple crop by their wealthy and fad-susceptible cousins to the north.

  The juice of the açaí is indeed lovely, but any firm claims about its healing properties are as nonsensical as those medieval totems that pilgrims traveled vast distances to touch for their curative powers. Basically, the deep belief that people have in the benefits of fruits like these, or of herbal remedies, is just a modern expression of those same inner desires. Similarly, as with the priests of old, someone is around to capitalize on this fervor to make a few bucks. At least the medieval clergy left us cathedrals; I have no idea what lasting monuments are being generated by the herbal supplement crowd.

  However, medications or approaches that lead to a completely new paradigm for a disease often don’t require such large numbers of patients to be studied precisely because their effects are so dramatic. The drug Gleevec, mentioned in a footnote earlier in the book, comple
tely changed the landscape for a fairly uncommon cancer known as chronic myelogenous leukemia, or CML. CML is a cancer that can be fairly indolent for years but ends in a deadly “blast crisis,” and treatments designed to stave off the blast crisis had only limited benefits and came with toxic side effects. Gleevec didn’t rid the body completely of the leukemia but it increased survival by several years—in some sense “curing” the disease, because CML patients are usually older anyway, so those extra years mean that CML patients pass away at around the same age as the average person. One of the famous early trials comparing Gleevec to the then-gold-standard treatment enrolled more than one thousand patients, but you didn’t need a PhD in statistics to see how superiorly Gleevec had performed: about 87 percent of patients taking Gleevec had a “major cytogenetic response,” compared to about 35 percent in the standard treatment group. One thousand patients weren’t needed to show that level of benefit. (NB: the “major cytogenetic response” is a commonly accepted indicator of treatment success, but, to be clear, the trial didn’t measure actual mortality, which is a more difficult thing to study in CML, owing to its prolonged course.)

  In my own field of infectious diseases, Hepatitis C treatment has, like CML, undergone a paradigm shift in the past several years. For years, the backbone of Hep C treatment was a drug called interferon alpha, which like the existing treatment for CML was both toxic and not especially effective. (This is because the backbone of treatment for CML also was interferon alpha!) The published cure rates for Hep C using interferon-based regimens were often only 50 percent, and in the real world this almost certainly meant that all patients had a lower chance of cure. Because interferon often caused patients to be very ill, the 50 percent success rate was based on patients enrolled in studies, and these patients are typically much more motivated to complete a treatment course, side effects be damned.

  Then in the late 2000s a few new drugs such as boceprevir and telaprevir came out, and suddenly the eradication of Hep C went from 50 to about 80 percent. Interferon was still required, yet real cure seemed a possibility for many more people. But the biggest change came in 2013, when the results of a trial known as COSMOS were presented at a professional meeting of hepatologists and infectious disease physicians. COSMOS involved a treatment regimen that didn’t require interferon, and although the trial did not compare these two approaches head to head, the cure rate with this new combination was about 90 percent. The study looked at only about 150 patients, and thousands won’t be required to see how profoundly our approach to this disease will be altered. By the time this book comes out, several new Hep C treatments that do not require interferon will be on the market, and most observers expect the cure rates of these to be so high that we are unlikely to know which of these combinations of drugs is the most effective, because at that point if drug X is 88 percent effective and drug Y is 92 percent effective, a trial to prove beyond doubt drug Y’s superiority would require tens of thousands of patients. So issues like cost, side effects, and raw business negotiations will determine which of these new drugs will be the most commonly used in the years to come.

  Much of the research discussed in this book looks at the blue marble / red marble type of studies. Blue versus red marbles in actual statistical language are known as categorical variables—the thing being studied is either this or that, either alive or dead, either cured or not. But many studies also look at continuous variables, which are measured in gradations. When we do research that looks at continuous variables, we aren’t pulling blue and red marbles from two jars, but rather marbles that are white at one extreme, black at the other, with fifty shades of gray in between.* When we do experiments of this sort, what we’re trying to find out is whether one jar is significantly darker (or lighter) as a matter of statistics, and that the difference in shade we’re observing is not due to chance.

  Couldn’t resist.

  For instance, hypertension, diabetes, and depression are all diseases that can be studied by evaluating their severity on a continuum. Suppose we want to know that a new diabetes drug does its job and actually lowers blood sugars. For this experiment we’d have our two jars (treatment and control) filled with thousands of marbles of various gray shades; darker shades represent higher blood sugars, lighter shades represent lower blood sugars. If the drug doesn’t work (our null hypothesis), we’d expect the average shade of the marbles we pull out of the treatment jar to be more or less the same as the average shade of the marbles from the control jar. By contrast, if the drug really works, we’d expect the average shade in the treatment jar to be lighter, and so much lighter that it couldn’t have just happened by chance.

  To avoid a lengthy departure into a discussion about standard deviation, standard error, and other statistical points, I won’t explain how the average shades are compared for differences. It’s easier to grasp how categorical variables are measured without doing much math beyond talking about coin flips. Suffice it to say that there are equations that enable this comparison making, and that their general outline is reasonably similar to the kinds of math required for evaluating categorical variables. To think about this in a nonmathematical way, it should be clear that if there’s a big difference in what’s being studied, fewer subjects are required to demonstrate the difference. The more subtle the difference, the more marbles must be taken from the jars.

  Clinical research on the effectiveness of antidepressants of the kind mentioned in the chapter on drug trials requires analysis of continuous variables like the Hamilton Depression Scale and its ilk. To review, the Hamilton Scale looks at a variety of psychosocial factors, assigns numbers to these factors based on the severity of symptoms, and adds them up to produce a number. Drug trials evaluating antidepressants compare the scores on depression scales of patients receiving the actual drug to those of patients receiving placebos. (They can also compare pretreatment to post-treatment scores, in which patients serve as their own controls.) It’s worth recalling that, when all the published studies on antidepressants are pooled together, the drop in depression scales is modest. Like the studies that evaluated high blood pressure, the biggest benefits probably accrue to the most severe patients, where a three- or four-point swing in a depression scale might reflect the difference between being able to get out of bed or not.

  The effect of prolonged antibiotics on patients with chronic fatigue syndrome, whether due to Lyme disease or anything else, is another clinical question addressed with continuous variables. In this case various “fatigue scales” are used in place of depression scales. No well-designed trial has ever found any benefit to this approach, although the chronic-Lyme ILADS enthusiasts would say that I have shown that I am a shill for the insurance industry for saying this.* Anyway, that’s how the studies would be done—and, indeed, they have been done. Chronic fatigue is awful, and most doctors feel helpless in the face of it, precisely because of the fact that we haven’t found any effective treatments thus far.

  Note to insurance companies: please send money, lots of money, for my advocacy. Though, sorry for the Hep C drug costs and all.

  Continuous and categorical variables can be combined as well. One of the early studies on smoking was published in 1952; it looked at 1,357 men who had lung cancer and compared them to an equal group of men who didn’t. Then they pulled the marbles from the jar and looked at whether they smoked. If they had looked at smoking only as a categorical variable (i.e., black or white marbles), they would have found that smoking was associated with lung cancer: 1,350 of 1,357 men with cancer smoked, while “only” 1296 men who didn’t have cancer smoked.† But they also looked at how much they smoked, and the results at the time must have been shocking, for smoking more cigarettes was clearly associated with cancer as seen on the next page:

  Keep in mind that this is a study of correlation, and that although the study found a significant correlation between smoking and lung cancer, as we discussed earlier that does not prove that smoking causes lung cancer. Indeed, the father of mo
dern biostatistics, Sir Ronald Fisher, used the “correlation doesn’t prove causation” argument to deny the importance of smoking as a risk for lung cancer well past its expiration date, going to his grave in 1962 as a chain smoker, presumably coughing all the way.

  In other words, the average grey shade of the marbles drawn from the cancer jar was distinctly darker than those emerging from the noncancer jar, not unlike the actual lungs these metaphorical marbles represent.

  But what is the use of all this talk of marbles and coins when the headlines come streaming across one’s electronic device,‡ or when a physician is talking about guidelines for screening or treating a disease? I would say that there are a few simple takeaways. One doesn’t need to have a PhD in biostatistics to get the basic idea of how relevant or how speculative medical research can be, and whether one should alter one’s life just because of the latest journal article or new set of guidelines.

  Or, for the hundred or so people who do not acknowledge the twenty-first century, the newspaper.

  First, know the type of research being done. Clinical research—that is, experiments done involving human beings—is always more relevant than any other form of scientific inquiry when it comes to health news. As I completed the first draft of this manuscript in August 2014, I saw a CNN headline with the title “Venom May Hold Cure for Cancer.” The article described the research of a scientist working with bee, snake, and scorpion venom, and how some of the proteins extracted from those fluids appeared to halt the growth of some cancer cell lines in a culture dish.