Snowball in a Blizzard

by Steven Hatch

  A fantastic book. An abysmally boring, stuffy title.

  This characterization can be found in popular culture as well. A few years ago the BBC aired a medical drama for two seasons. Known as Casualty 1907 and Casualty 1909 and marketed outside the UK under the title London Hospital, the show was a carefully constructed imagining of what life was like as modern medicine was taking shape in earnest. As much as the show was meant to entertain, it also clearly envisioned itself as a form of dramatic history lesson, in effect asking its viewers to think about how much has changed, but also what has not. We see, for instance, a rigid sexual hierarchy that has since been (mostly) obliterated, with male surgeons and physicians dashing about in dapper Edwardian dress, giving unambiguous orders to female nurses clad in demure floor-length dresses, color coded to their level of rank. We follow the patients’ stories as they lie in large public wards instead of private rooms, many of them dying of diseases that we now dispatch with a spritz of penicillin. On the surface, it’s a very antiquated environment.*

  While I was writing this book, an American TV show called The Knick aired on the network Cinemax, which was similar in its preoccupations.

  But those familiar with the inside of a hospital will find some of the similarities to today’s health-care facilities uncanny: the aseptic technique of the OR, with gowned, gloved, and masked personnel, is practiced; infectious outbreaks, despite the inability of the staff to use antibiotics because they weren’t yet discovered, are monitored and rapidly quarantined; and a variety of what was then experimental scientific gadgetry is employed, the clear forerunners to our high-tech medical subspecialties such as radiology. Their technology wasn’t as sophisticated as ours, but these doctors and nurses, and the medical system they inhabit, is recognizably modern. They know what they are doing, at least in broad outlines. Moreover, they know what they know and they know what they don’t, and that there is more to be discovered in the years to come. You can almost sense they are aware that modern doctors and nurses will be looking back at their work, knowing it was unsophisticated at one level but also aware that such work was on a trajectory. We are like you, these characters whisper. We have solved the puzzle about how to know. It’s a matter of details from here on out.

  Those characters, although invented in a contemporary writer’s head, are saying something true about early modern medicine. We really can draw a straight line between us and them; their tools were crude, but we approach patient care and think about pathology in fundamentally the same way. The arrow of medical and scientific progress is quite real.

  I work as a physician and was educated in this scientific method in the manner of tens of thousands of my brothers and sisters over the past century. We were trained in places like Iowa, Addis Ababa, London, Tokyo, and Mumbai. We speak a common language and have similar ways of thinking such that I can travel to Monrovia in the heart of West Africa, get off the airplane, go straight to the hospital and evaluate a patient there, offer drugs from their stockroom with which I am familiar, and teach nascent doctors about disease, in much the same way that I do in Worcester, Massachusetts. And I know that what we provide with our so-called Western approach can have a much more significant impact on the diseases people face in all of those places compared to the offerings of those who still traffic in folk remedies.

  Yet, like all characterizations rooted in a powerful truth, our pride in our modernity has the potential to blind us to our own shortcomings and leave us overconfident in our abilities.

  This book is in large part about those shortcomings and the resulting overconfidence it can produce. The term we’ll give to this phenomenon is uncertainty. In the coming pages, we’ll carefully consider uncertainty—specifically, the uncertainty that permeates the theory and practice of modern medicine. The book’s premise is simple: namely, that doctors do not often “know” what they are doing with the same kind of mathematical precision that we associate with rocket scientists or chemical engineers. A diagnosis is, much more often than not, a conjecture, and a prognosis is typically less certain than that. There is a good deal more haziness in the world of medicine than most people—those both outside and inside that world—understand. The consequences of those misunderstandings can be perilous for physician and patient alike.

  Uncertainty lies at the heart of what physicians do on a daily basis. Sometimes they are entirely aware of it, and sometimes they fail to appreciate it. Sometimes it prominently features in discussions between doctor and patient. And sometimes it is completely misunderstood. The purpose of this book is to show the reader not only that this is so, but how it is so as well.

  Many of the original thinkers on probability and uncertainty were card playing and gambling types living in the eighteenth and nineteenth centuries. This isn’t accidental, as these pastimes predispose one to bend one’s thinking toward the statistical. It would take medicine a few centuries to catch on in earnest, but the groundwork for incorporating uncertainty into medicine was being laid during this heyday of the Enlightenment. Today, the early deeds of these medical pioneers are typically intoned with great solemnity at some occasion involving pomp and circumstance such as a White Coat ceremony or a medical school graduation. Interestingly, such evocations of the past are done for almost precisely the wrong reasons, with the protagonists being falsely depicted as bringers of truth and light to otherwise ignoramical colleagues. In Chapter 6, we’ll see one of the most famous examples of a great medical hero who is typically portrayed as a towering genius, only he misunderstood the meaning of the very discovery he was credited with making.

  Much of this book will discuss uncertainty by emphasizing the underestimated imperfection of results. My goal will be to show that these results, whether those of an individual blood test or those of a 10,000-person study five years in the making, need to be approached with varying levels of caution. I will try to highlight some areas in which doctors or patients or both have gotten themselves into trouble by neglecting uncertainty when they interpret results, not realizing that a positive test may sometimes be negative in reality or that a new miracle drug may not be so miraculous.

  In the coming pages, I will attempt to survey the landscape of uncertainty in the diagnosis and treatment of human disease. One central assumption I make is that uncertainty, at least for the foreseeable future, is an irreducible feature of modern medicine and that understanding uncertainty is a vastly better strategy than ignoring it. My aim here is to explain those areas in which medical problem solving is most profoundly misunderstood, precisely because such misunderstandings can have, at the extreme, lethal consequences. This is as true for the physician who blithely and injudiciously prescribes a course of antibiotics for an elderly patient with a touch of a cough, who subsequently develops severe antibiotic-associated Clostridium difficile colitis, as it is for the family members of a patient in the ICU who keep pressing the medical team to perform invasive, high-risk tests that aren’t likely to help with their loved one’s outcome. This is as true for the policy makers and “disease advocates” who recommend screening tests that sometimes aren’t very accurate as it is for the politicians who may take unscientific, and ultimately harmful, positions in the pursuit of currying favor with a special interest group. In short, I intended to make this book a practical exercise, a consideration of the consequences of uncertainty in medicine.

  You might be wondering right now how uncertainty takes shape—that is, what does it actually mean to say that doctors are either uncertain about what they are doing or are overly confident because they haven’t taken enough uncertainty into account? To better acquaint ourselves with how uncertainty manifests itself, let’s consider one of the most well-known doctor-patient scenarios in medicine: the “cancer prognosis” talk. After all, when newly diagnosed cancer patients sit down with their oncologists, they ask a reasonable question: how long do I have to live? Most of us would expect to hear a dispassionate prediction from the physician as they stare the patient squarely, if sympat
hetically, in the eyes: I’m sorry, but you have 8 months . . . or you have 2 years or some other hard number that will coldly and scientifically state the simple truth.

  What moment in the physician-patient encounter could be more well-known? This conversation forms the basis of plot lines in TV dramas and movies. Many or most patients and their family members rightly assume that, given the staggering array of blood tests and body scans that are performed in the aftermath of a new cancer diagnosis, all of that information can be reviewed by an oncologist and lead to a fairly accurate prediction of survival time. Nobody thinks that oncologists can predict someone’s remaining time to the day or the week, but most assume that their predictions are accurate to within at least a few weeks’ time.

  In fact, oncologists almost never make these kinds of predictions because, as a rule, they’re not very good at them. Only as death approaches closely do oncologists become reasonably decent at prognosticating survival length—and even then, the evidence that they predict survival time accurately is mixed at best. One review found that, even among terminally ill patients whose median survival is only four weeks, doctors were correct to within a week of survival in only 25 percent of cases, and in another 25 percent their predictions were wrong by more than four weeks! This review paper looked only at patients who were clearly at the end of their lives, and pretty much anyone, whether they possess a doctorate in medicine or not, can look at such patients and make a prediction with the same level of accuracy. So oncologists are keenly aware that guessing the life span of a patient with virtually any cancer, unless they are presenting at a very advanced stage, is an exercise in folly.

  What oncologists can do with much greater accuracy is talk about the behavior of groups of people who have a given cancer that present at a given stage. Based on data collected about cancer patients over the past four decades, they can talk about the odds of survival. For example, we know that a patient who has localized bladder cancer has about a 70 percent chance of being alive at five years. We know this because cancer is a disease that is tracked by the federal government—physicians are required to submit each case to a national database we’ll explore later—so that number is fairly precise. But oncologists saying to patients that they have a 70 percent chance of survival at five years is a very different thing than predicting they have about four years left of life, as some patients with bladder cancer will decline very quickly, and others will live for many years to come. Such discussions necessarily entail an honest admission by clinicians that they cannot look into the crystal ball, and such statements are only meant for patients and families to consider the odds, weighing the risks and benefits as they move forward and make decisions about their care because cancer treatment can often make patients very sick and reduce their quality of life.

  Even here, however, the acknowledgment that a patient is subject to laws governed by probability rather than certainty can sometimes prove misleading. If a patient has squamous cell cancer of the lung, a very common kind of cancer, and the cancer is staged accurately, a doctor’s statement that the patient has a 40 percent two-year survival with aggressive treatment is likely to be very accurate. This is because thousands of people each year develop this disease, and data from such a large cohort is less subject to the vicissitudes of random statistical fluctuations. Thus, researchers can know with reasonable precision how many people are likely to survive in a given time span.

  But take a more unusual cancer, such as chondrosarcoma. This disease, a cancer of cartilage cells, is quite uncommon: only about four hundred people are diagnosed with the disease each year in the United States. Moreover, chondrosarcoma strikes people at various stages in life, and the cancer can appear at different parts in the body. It may turn out that an overall 40 percent survival is simply because the average of the past two years was 10 percent followed by 70 percent. Thus, the rarity of a given disease can cause even confident statements conceding inherent uncertainty to be untrustworthy! This is uncertainty in action. But it can be found everywhere in medicine, not just cancer diagnosis or prognosis. My goal here is to introduce you to some of the most important medical topics today in which uncertainty plays a starring role.

  Snowball in a Blizzard

  I chose the title Snowball in a Blizzard in part because it provides a useful metaphor for uncertainty. Picture a game in which we are testing you on your ability to recognize snowballs thrown through the air by some person, say, one hundred feet away, in the midst of a raging blizzard. You don’t how many snowballs we’re going to throw nor how often nor how fast or slow. You just have to look out into the whiteness and decide whether you see randomness or you have identified something as worthy of attention.

  It should not be too hard for readers to picture the difficulty in the task. In the first chapter, we’ll see an example where a scientist fiendishly performed almost exactly this experiment, except instead of using snowballs, he used schizophrenics while sane individuals served as the blizzard, and he tried to see whether psychiatrists were, as people would generally assume, good at spotting the “snowballs.” (Though, to be clear, he didn’t throw the patients through the air but rather had them present to psychiatric hospitals for admission.)

  Snowball in a Blizzard, then, underscores that uncertainty is a structural component of data interpretation and is not merely some occasional and accidental feature of the system. Sometimes the uncertainty lies in diagnosis: Does this test really mean that I have this disease? Sometimes it pertains to treatment: If I take this drug, am I really going to benefit from it? Sometimes it concerns environmental risks: Is it really okay for me to have coffee while I’m pregnant? Rarely are the answers to these questions a simple unqualified yes or no. Uncertainty is nearly always part of the discussion; the only real question is, to what extent?

  Moreover, Snowball in a Blizzard has a special resonance in medicine, for it is a well-known phrase among one group of doctors, a sort of inside joke they have indicating their keen appreciation of the complexities of data interpretation. I learned of it many years ago when I was a medical student at the University of Cincinnati, when I was rotating on the radiology service. One day we attended a lunch sponsored by the department, intended to be an overview for any of us who might be interested in pursuing radiology as a career. One of the speakers was finishing up his fellowship in pediatric radiology, and he had just accepted a position at a suburban hospital outside Philadelphia. “I’ll be doing mostly general radiology, a little bit of everything,” he said, but quickly added, “though I’m not going to do mammography. They have other folks for that. And I’m perfectly happy to avoid mammography anyway.”

  “Why would you want to avoid mammography?” someone asked.

  “Because it’s like trying to find a snowball in a blizzard,” he immediately replied.

  The phrase hit me like a thunderbolt. I came to learn that the witticism wasn’t his originally but was a bit of grim humor passed around by radiologists as a commentary on the difficulties of detecting breast cancer—the “snowball”—in the “blizzard” of otherwise healthy breast tissue. (We will have much to say about mammograms in the coming pages.) Many radiologists, this doctor noted, have found themselves facing lawsuits for having missed tumors in women who went on to develop cancer.

  Thus, “snowball in a blizzard” compresses all of the challenges of uncertainty into one pithy phrase. In doing so, it also expresses something peculiar about medicine that is not quite the same as the uncertainty discussed in other recent books, such as Nate Silver’s The Signal and the Noise or Nicholas Nassim Taleb’s The Black Swan. Those books have tackled, for instance, the problems associated with guessing which baseball team or political candidate will win, what will happen with the stock market, or when the next big earthquake will hit California. These are unknown future events, and the bare thesis of these books might be thought of as predicting the future is not impossible, but it’s more difficult than you think. Yet, when radiologists make cracks about
finding snowballs in blizzards, they are implying not only that the future is uncertain but that knowing what’s going on right now and directly in front of one’s nose can be equally uncertain!

  Finally, it is fortuitous that the phrase “snowball in a blizzard” is used specifically with reference to mammography, for not only does it describe the technical challenges involved in accurate readings of mammograms, it also serves as a metonym about the contentious debate that has evolved around the practice. The biggest killer in the Western world is, by far, cardiovascular disease, and yet it is mammography that is arguably the most hotly debated medical technology in public health policy, particularly in the United States. Much has been written about mammography and the dimensions of the public discussion; my goal in a later chapter will be to apply a small amount of mathematical rigor to the debate to clarify the logic that guided the public health authorities when they issued new guidelines on screening mammograms several years ago.

  The Spectrum of Certainty

  I will turn throughout the book to the notion of the spectrum of certainty—just how much we know about a given subject—and then make comparisons with other health matters. It is a compass by which one can navigate the landscape of doctorspeak and the weighty decisions doctors or health authorities sometimes ask patients and family members to make. For instance, we know with a great deal of certainty that a sedentary lifestyle combined with a high-calorie, high-fat diet puts an individual at high risk of a variety of unpleasant medical problems. But do we know whether eating dark chocolate once each day will help prevent Alzheimer’s dementia? As I will show in Chapter 7, the answer is, not so much. But let’s sketch out the spectrum and then step back and see how it can be useful.