The Danger Within Us

by Jeanne Lenzer

  Using games of chance to illustrate his point, he said, “Any one person’s chance of winning the lottery is virtually zero, but someone ends up winning it every time. Waiting for a particular extremely unlikely event is almost certain to be disappointing. But of the infinite number of extremely unlikely events out there, some—an infinite number, actually, if you want to be mathematically precise—do actually happen.”

  Hoffman’s example helps to explain the medical illusions created by the five patient and family-member testimonials in the FDA hearings. Each was powerful in the telling, and each was resoundingly positive. While individual patient stories are important and can lead doctors to new discoveries, they need to be understood in context. If a treatment works fabulously in only ten out of a million patients, but those ten are the only ones to testify, we would get a distorted picture of that treatment. In the same way, if the VNS device works really well for most patients and causes problems in very few, then Fegan’s quest to have it removed from the market would be akin to a patient’s family demanding that penicillin be taken off the market because a family member died of an allergic reaction to the drug. Context, and the overall comparison between harm and benefit, are essential.

  Unfortunately, all the statistics in the world can be a poor cousin to the power of anecdote and testimonials. Studies show that doctors as well as patients fall under the sway of anecdote.232, 233 While the glowing testimonials offered at the 1997 FDA hearing were impressive enough individually, the fact that all five testimonials, including one from the director of the Epilepsy Foundation, were positive might seem to suggest that there is no question about the safety and efficacy of the VNS device. But that would be an illusion.

  One of the powerful forces driving illusions in medicine is money, which not only biases research design and reporting but also influences patient testimonials. The patients and family members who addressed the panel that day were not chosen at random from the entire pool of people implanted with the VNS device. They were not the injured, as Dennis Fegan was. Nor were they the family members of patients who died after implantation. Instead, each patient or family member was carefully selected. And their airfare, food, and hotel accommodations were paid for by the Epilepsy Foundation, which, as Paulette Machara, the foundation CEO, acknowledged, did receive funding from Cyberonics. After each member of the public spoke, he or she was asked, as required by the FDA, to reveal any financial conflicts. The disclosures were all exactly the same as this interchange between patient Robert Cassidy and the panel chair, Harold Wilkinson:

  DR. WILKINSON: And again, your way was paid by the—

  MR. CASSIDY: My way was paid by the foundation.

  DR. WILKINSON: And not by the company.

  MR. CASSIDY: Not by the company whatsoever.

  This is routine at FDA hearings: the manufacturer of whatever device (or drug) is being reviewed pays a foundation or “patient group” in order to put its commercial message in the mouth of a seemingly independent third party. Then the third party, either alone or with the help of the manufacturer, selects patients to testify who will promote the product under review by the FDA. It’s a well-worn path of high farce: industry effectively launders its money through third-party organizations, and everyone acts as if it isn’t happening. *

  The process is akin to Congress inviting only the winners of million-dollar lotteries to testify about the “benefits” of gambling. Losers are not invited. Selection is everything, and as the saying goes, “Dead men tell no tales.” What is not reported is equally as important as what is reported.

  This doesn’t mean the patients were dishonest—not at all. They were the lottery winners, and they were honestly describing their good fortune. But whether a patient’s good luck is indeed attributable to the device or to luck is another matter. While it’s possible that the VNS device has led to improvement in a subset of patients, it’s also possible that their improvement was the result of other factors altogether.

  The play of chance, and our human desire to find meaning in whatever we experience, takes other forms that can mislead us. Hoffman described another example: “My very first patient as a student on the medicine service noticed a lump in his armpit while reaching for something in his closet. The lump was actually a lymphoma, and of course it had been there for quite some time. But since he first noticed it when reaching, he connected the two in a way that made sense to him—he was sure the lump meant that he ‘must have strained something.’ That allowed him to impose some sort of order and rationality on the world. In fact we all do that, all the time, and it usually helps us understand our experiences. But not infrequently it also leads us to conclusions that are simply wrong.”

  Like Hoffman’s patient who connected the lymphoma under his arm with the action he was taking when he found it, patients who improve following implantation with a device are bound to believe the device caused the improvement. Yet studies show that a sizable number of patients with temporal lobe epilepsy (the form associated with partial seizures) undergo spontaneous remission. They simply stop having seizures for reasons unknown. More than half of 190 epilepsy patients who were followed for a mean duration of eleven years underwent spontaneous remission.234 Another study, of sixty-four patients followed for a median of thirteen years, found that nineteen of them (31 percent) attained a five-to-fifteen-year remission and were able to remain off treatment without seizures.235

  Another factor contributing to the illusion of benefit is the placebo effect. Patients will provide testimonials that they were cured of cancer by the laying on of hands or that coffee enemas cured their arthritis. It’s not that these patients aren’t honest—or didn’t improve. In some instances, they improved dramatically. But the power of belief, chance timing, and the natural fluctuations of illness and spontaneous regressions that occur with almost all diseases—even some deadly cancers—can lead to potent illusions.

  The placebo effect is no less powerful with implantable medical devices than with drugs. During the hearing to approve the VNS device, FDA adviser Piantadosi commented on the fact that one of the measures of benefit claimed by Cyberonics—the total number of seizures experienced before and after implantation—could not control for the placebo effect because test subjects knew they had the device and all believed they might benefit. A real demonstration of benefit could come only from a differently designed study involving a medically managed control group.

  Another key to the prevalence of medical illusions is the fact that not everyone is using the same yardstick to measure results. As Hoffman told the physicians at Bellevue, the word benefit, like the word significant, has a meaning in medical research that is entirely different from its meaning in common usage, and that difference frequently misleads doctors and patients alike. When used in everyday language, benefit is assumed to mean net benefit. If someone says she made a financial investment that was beneficial for her, we understand that the return on her investment was greater than any charges or fees she paid.

  That isn’t the case in medical research, where benefits are defined and reported separately from harms. In some ways that’s for a good reason. If a treatment is intended to reduce heart attacks, and it does so, but it also causes liver damage in some patients, there is no single outcome measure that can reliably capture the true picture of the treatment. But this approach has opened the door to manipulation by sponsors who may test multiple end points and use multiple measuring sticks, then report only the findings they like. Frequently the study design itself prevents the full picture from emerging.

  Similarly, Cyberonics used an isolated measure of benefit to declare the VNS device a winner. They defined benefit as the percentage of test subjects who had at least 50 percent fewer seizures after implantation. As for test subjects with an increased number of seizures? Well, that percentage was simply ignored in the calculation of benefit, because that’s not part of the company’s definition of benefit, or effects caused by the device. Instead, individuals who ha
d more seizures were listed in small print under side effects. In this way, Cyberonics could claim in promotional materials that approximately a third of individuals obtained benefit from the VNS device.

  Hoffman cites another example that illustrates widespread misunderstanding about the word benefit. Deaths from cancer are typically reported as “disease-specific” deaths, meaning that the benefit of a screening program for lung cancer is measured by the decrease in lung cancer deaths after screening. At first blush, that seems reasonable. Isn’t that what everyone wants to know, after all? Do screening and treatment for lung cancer reduce deaths from lung cancer?

  But as Hoffman points out, relying on disease-specific mortality instead of overall or all-cause mortality completely ignores the potential dangers of diagnosis and treatment. Even if screening for lung cancer prevents some patients from dying of this cancer—and thus decreases the disease-specific mortality—we can’t know if screening is worth doing unless we also know whether it leads to just as many or more patients dying because of the screening program, whether from an invasive diagnostic procedure, or from surgery, or chemotherapy, or radiation. And that’s only mortality: we also need to compare likely harm and likely benefit in quality of life, not to mention cost.

  Despite the common perception that screening can only be beneficial, there are many documented examples of overall harm from screening. Hoffman tells us about one of these:

  In the 1970s a device was developed that could accurately detect neuroblastoma, a deadly cancer affecting children. The device, used to test the urine of infants and toddlers, could pick up the cancer at a very early stage, long before children developed any symptoms. However, the test was abandoned after it was discovered that babies screened with the test were actually slightly more likely to die than unscreened babies.236–238 It turns out that many of the tiny early cancers that were causing no symptoms, and therefore were only detected by screening, were very likely to regress spontaneously—so no one would have known about them if the babies hadn’t been screened. But because doctors and parents didn’t feel comfortable doing nothing, they treated all babies—not just babies with symptoms who actually required treatment. Unfortunately, the program led to the detection of so many of the early cancers that a great many children were exposed to the potential perils of surgery—from which almost none of them could possibly benefit, since almost all these cancers would have simply gone away by themselves. In the end, the number of lives saved by treatment was exceeded by the number of lives lost because of the screening program.

  According to Hoffman, doctors frequently fail to fully appreciate how the benefits of a treatment change with the severity of the condition being treated. He explains the concept, which he originally described in “Overdiagnosis of disease: A modern epidemic,” published in 2012 in the Archives of Internal Medicine:

  Imagine a new antibiotic; we’ll call it “gorillacillin.” The drug is so toxic that it kills ten percent of those who receive it—after all, it’s a very powerful drug that is tremendously beneficial for patients with the dread disease, “infectiosis,” decreasing death rates from fifty percent to twenty-five percent. Gorillacillin is less attractive, however, when only 20 percent of treated patients actually have infectiosis; the 10 lives saved among the 20 patients who would have died are completely offset by the 10 drug-related deaths among 100 patients treated.

  Now imagine that all 100 patients actually have infectiosis, but 90 percent were diagnosed by sophisticated tests performed despite the absence of the classic fearsome symptoms of infectiosis, “just to be sure”—such that precious few have the deadly form of the disease. Since virtually none of these 90 were at risk of dying, gorillacillin would save 5 of the 10 truly at risk, but by killing 10, it would cause net harm. Overdiagnosis inevitably means that many individuals are subjected to the potential harms of treatment while being afforded almost none of its benefits.

  The example of gorillacillin is counterintuitive; it flies in the face of conventional “better safe than sorry” wisdom and our belief that catching something early is “a stitch in time.” But with the new screening test for “infectiosis,” a different and far larger patient population is treated—a population in which many patients will recover without any medical intervention. Exposing this far larger population to the harms of gorillacillin now increases the chance of death.

  Illusions and misunderstandings like these go a long way toward explaining how a highly questionable device like the VNS can get approved and even become a widely accepted treatment.

  Common marketing tactics used by device manufacturers take advantage of medical illusions to increase the allure of their products. One such tactic is the indirect suggestion of a health benefit that the device doesn’t actually provide.

  Cyberonics’ promotional materials, videos, and website feature patients who say that thanks to the VNS device, they were able to decrease or stop their medicines—freeing them of drug side effects. The belief that they would be able to reduce or stop medicine attracted many patients to the device. During the FDA hearing, Patricia Kroboth said that what excited her was the chance to get her son off medicines. And Paulette Machara of the Epilepsy Foundation implicitly suggested that stopping or reducing medicines would be an advantage of the VNS device when she said that it would benefit pregnant women because it “wouldn’t harm the developing fetus.” Yet in clinical trials of the VNS device, patients did not reduce their medicines.

  Another unsupported, but implicit, claim by advocates is that the VNS device would save lives. Since individuals with epilepsy are known to have higher death rates than the individuals without epilepsy, advocates have emphasized the death rate when they promoted the device. But far from saving lives—there was evidence suggesting that patients implanted with a VNS device might be more likely to die—evidence that led the FDA to grant it only conditional approval.

  A careful look at Cyberonics’ claims of benefit reveals something quite different from the rosy picture painted by the company. Cyberonics’ claim was based on the percentage of subjects in studies E03 and E05 who had a 50 percent or greater reduction in their seizures over baseline (as measured during the twelve weeks prior to enrollment in the study).120, 135 According to the company, 31 percent of the high-dose group in E03 achieved a 50 percent or greater reduction in seizures, while 13 percent of the low-dose, or sham group did the same. The difference between the two groups was the best indicator of benefit, since the sham group’s benefit was expected to be due to placebo effect. In other words, only 18 percent (31 percent minus 13 percent) of test subjects achieved a 50 percent or greater reduction in seizures. Cyberonics claimed in promotional materials that one-third of patients improved (based, apparently, on the 31 percent who improved in the high-dose group). Regardless of whether one accepts the 18 percent or 31 percent rate of benefit, this means that 69 percent to 82 percent of patients achieved no benefit whatsoever from the device—and indeed, as Fegan discovered while reading Cyberonics’ data provided to the FDA, one-fifth to one-third*of individuals actually had more seizures.

  In study E05, Piantadosi noted, the number of seizures experienced by test subjects was “highly skewed,” meaning that while some patients had fewer seizures, others had many more seizures—as much as 690 percent more. Piantadosi said the true measure of benefit should be median improvement, and in that measure E05 showed no benefit. But Cyberonics used a different statistical measure to claim a benefit, one that nonetheless teetered on the edge of statistical significance.

  The effect of the VNS device on seizures was like what happens when you squeeze a balloon in the middle—it caused the balloon to become bigger at the top (representing patients with fewer seizures) but it was also a lot bigger at the bottom (representing patients with more seizures). Cyberonics was effectively reporting only the top of the balloon without reporting the bottom to make its claim that the VNS device benefited a third of patients.

  There were a number of other
problems with Cyberonics’ study design. During deliberations, the FDA’s Ann Costello remarked that the company’s claim of benefit in the one-year extension phase of study E05 was eroded “by the fact that the patients were changing their medications during this period.” Had improved medication management contributed to the marginal benefit? Costello’s point is one that dogs many device studies: there is often no control group of patients treated with medicines only and without a device. Failure to include a medical control group has led to many unnecessary device implants, from elective coronary stent placement to inferior vena cava filters—both of which were found to offer little to no benefit over medical management.129, 239

  By failing to include a medical management arm of the study, Cyberonics set up a win-win situation, because one arm was bound to do better than the other. If the high-dose group performed better than the low-dose group, that was a win, and if the low-dose group proved superior, it would be possible to say that the high dose was simply too high. (Note the company’s language—the doses were “presumed” therapeutic versus subtherapeutic, thus leaving wiggle room for just such a claim.) The study design made it impossible for Cyberonics to lose.

  Nor could safety outcomes be accurately assessed, because Cyberonics excluded patients from enrolling in the studies if they had medical problems such as lung disease, heart disease, peptic ulcers, or status epilepticus. Certain later studies supported by Cyberonics claimed to show that the device could reduce deaths, but it did so by comparing study patients, which excluded the sickest seizure patients, and compared their death rate to historical control groups making interpretation virtually impossible, since reaching back to death rates among patients in the past ignores a host of confounders: Were patients in older timeframes treated with less effective medicines than they would be today? Were they a different age group than patients selected for VNS implants? Is it possible that VNS patients did better simply because they were less sick to start with?