by Ian Ayres
Other physicians reject evidence-based medicine, saying that there often aren’t any good statistical studies to guide their decision. Even today, many medical procedures do not have systematic data to support them. Lisa Sanders, the “Diagnosis” columnist for the New York Times Sunday Magazine and an internist at Waterbury Hospital in Connecticut, puts it this way: “Even though evidence-based medicine has made tremendous strides,” she says, “still only a tiny fraction of what we do in the course of a day is actually evidence-based.”
Lisa is the perfect person to step back and think about the rise of evidence-based medicine, because she has lived through the sea change in emphasis from the early days of her training in the 1990s. “Statistics really wasn’t taught until a few years ago,” she says. “I was on the beginning of the wave.”
Born and raised in South Carolina, Lisa is a true renaissance woman. In a former life before becoming a doctor, she was an Emmy-winning producer for CBS news. Her current “Diagnosis” column is the inspiration for the hit TV show House, where she is the lead medical consultant. She cares passionately about evidence-based medicine, but sees its limitations.
“Right now I am teaching all of these residents all the stuff of the physical exam,” she tells me with characteristic candor, “but I don’t tell them how sensitive it is or how specific it is or what its positive predictive value is because mostly we don’t know. I try to—to whatever extent I know about it. But it is still a huge gray area, and when you think that one of your fundamental tools—the physical exam and taking the history—is not at all evidence-based at this point, it’s just kind of crazy. So that is where we are. We are just beginning.”
In some ways, it shouldn’t be surprising that there aren’t always evidence-based answers. Medical research is damned expensive. The cost not only cuts down the quantity done, it also biases what kinds of questions get studied and reported. Pharmaceutical companies fund efforts to prove that their drugs are effective, but are not about to ask whether non-proprietary treatments are equally as good.
And even when rigorous empiricism exists, it may not provide information that is relevant to the patient in question. The protocols for clinical trials routinely disqualify patients with many comorbid diseases so that the study can exclusively assess the impact of a treatment on a particular disease. And there has been an unfortunate tradition of excluding women and minorities from these trials.
It’s also reasonable for doctors to resist statistical studies that are badly done. Some studies don’t ask the right questions or control for sufficient variables. A few years back, a study concluded that excessive caffeine consumption increased the risk of heart disease—without controlling for whether the patients in the study smoked. Turns out that smokers were more likely to drink a lot of coffee and that smoking, not caffeine, was the real cause for the heart failure. This kind of resistance is completely consonant with evidence-based medicine. EBM only asks doctors to assess the quality of different types of evidence and to give high-quality, systematic research the appropriate weight when making their treatment decisions. The problem is that many doctors fail to make any assessment of relevant EBM results because they never learn that a study exists.
“You Could Look It Up”
A physician’s decisions can’t be driven by statistical results if the physician doesn’t know what the statistical result is. For statistical analysis to have impact, there needs to be some kind of a transmission mechanism that disseminates the analysis to the decision makers. The rise of Super Crunching is often accompanied and nurtured by improvements in the technology of transmission so that decision makers can more quickly access and respond to data in real time. We’ve even seen automation of this transmission link in randomized Internet testing applications. Google’s AdWords not only reports your results in real time, but it will automatically shift your web page image toward the version that produces the best results. The faster Super Crunching results are available, the faster they can change a decision maker’s choices.
In sharp contrast, the practice of medicine before the EBM movement was shackled with an extremely slow and inefficient mechanism for disseminating medical results. The Institute of Medicine estimates that it took “an average of 17 years for new knowledge generated by randomized controlled trials to be incorporated into practice, and even then application [was] highly uneven.” Progress in medical science occurred one funeral at a time. If doctors didn’t learn about something in medical school or in their residency, there was a good chance they never would.
As in other Super Crunching contexts, the EBM movement has tried to shorten the time it takes to disseminate important results. The central, but probably most deeply resisted, demand of EBM is the call for physicians to research the problems of their particular patients. Just as Semmelweis angered doctors by calling on them to wash their hands several times a day, the evidence-based medicine movement has had the impertinence to ask doctors to change what they do with their time.
Of course, doctors shouldn’t do patient-specific research for all their patients. It would be a huge waste of time to hit the books when a patient presents with the classic symptoms of a common cold. And in the emergency room, there simply isn’t time to do research. But academics who have “shadowed” practicing physicians find that about one in three new patients presents questions that would benefit from a review of current research. And this proportion rises for patients newly admitted to a hospital. Yet very few physicians in these studies actually take the time to look up the answers.
The critics of evidence-based medicine often focus on the lack of information. They claim that in many instances high-quality statistical studies just don’t exist to provide guidance for the myriad of questions that arise in day-to-day clinical decision making. A deeper reason for resistance is just the opposite problem: there is too much evidence-based information for any individual practitioner to reasonably absorb. On coronary heart disease alone, there are more than 3,600 statistical articles published each year. A specialist who wants to keep up in the field would have to read more than ten articles every day (including weekends). At fifteen minutes an article, that’s two and a half hours a day dedicated to reading about just one class of disease. “And it would be a waste of time,” Lisa says. “Most of those articles aren’t any good.”
Clearly, asking physicians to devote such a large amount of time to sifting through mountains of statistical studies was never going to be feasible. From the very beginning, the advocates of evidence-based medicine understood that information retrieval technology was critical to allowing practicing physicians to pull the relevant and high-quality information from the massive and ever-changing database of medical research. Indeed, in their original 1992 article, Guyatt and Hackett looked into their crystal ball and imagined how a junior medical resident might respond to “a 43-year-old previously well man who experienced a witnessed grand mal seizure.” Instead of just asking a senior resident or attending physician what to do, the evidence-based practitioner would proceed “to the library and, using the ‘Grateful Med program,’ conduct … a computerized literature search…. The search coststhe resident $2.68, and the entire process (including the trip to the library and the time to make a photocopy of the article) [would take] half an hour.” The idea that a resident could go to the library, locate, and photocopy an article in half an hour in retrospect was ludicrously optimistic, but something has happened since 1992 to make the idea of half-hour research much less of a pipe-dream: Al Gore’s friend, the Internet. It’s sometimes hard to remember, but the first web browser wasn’t released until 1993.
“It was technology—the computer, the Internet—that finally made it possible for the doctors in the trenches, doctors taking care of patients, to systematically practice evidence-based medicine,” Lisa says. “We all now have access to the newest and best in medical research right at our own desks. We can find out quickly, for example, that arthroscopic knee surgery for osteoarthritis an
d postmenopausal hormone replacement for the prevention of heart disease have lost their standing as effective therapies.” Because of the web, every examination room now can have a virtual library.
The retrieval technology of the web has made it a lot easier for physicians to find the results that relate to the specific problems of specific patients. Even though there are more high-quality statistical articles than ever, it is simultaneously faster than ever for physicians to find the haystack’s needle. A host of computer-assisted search engines now exist that are dedicated to putting doctors in touch with relevant statistical studies. Infotriever, DynaMed, and FIRSTConsult are all there to help doctors find the summaries of cutting-edge research with just a few clicks of their mouse.
These summaries of results usually have a web link so that the physician can click through and see both the underlying study and all subsequent studies that have cited it. But even without clicking, a doctor can tell a lot from the initial search result just by looking at the “Level of Evidence” designation. Today, every study is given a grade (on a fifteen-category scale developed by the Oxford Center for Evidence-Based Medicine) as a shorthand way to inform the reader of the quality of evidence. The highest possible grade (“1 a”) is only awarded when there are multiple randomized trials with similar results, while the lowest grade goes to suggested treatments that are based solely on expert opinion.
This change toward succinctly characterizing the quality of the evidence may well be one of the most important impacts of the evidence-based medicine movement. Now a practitioner in evaluating a study recommendation can have a much better idea of how much he or she can trust the advice. One of the coolest things about Super Crunching regressions is that they not only make predictions but are able to simultaneously tell you how precise the prediction is. The same kind of thing is now happening with these levels of evidence disclosures. EBM not only makes treatment recommendations but simultaneously tells the physician the quality of the data backing up the recommendation.
This grading of evidence is a powerful response to the naysayers who claim that an evidence-based approach won’t fly because there simply aren’t enough studies to answer all the questions that doctors need to answer. Grading still lets experts answer pressing questions of the day even in the absence of authoritative statistical evidence. It just requires them to reveal the limits of the field’s current knowledge. The level of evidence-grading scale is also a simple but real advance in information retrieval. The harried physician can now cruise through dozens of Internet search results and distinguish anecdotes from robust multi-study outcomes.
The openness of the Internet is even transforming the culture of medicine. The results of regressions and randomized trials are out and available not just for doctors but for anyone who has time to Google a few keywords. Doctors are feeling pressured to read not just because their (younger) peers are telling them to, but because increasingly they read to stay ahead of their patients. Just as car buyers are jumping on the Internet before they visit the showroom, many patients are going to sites like Medline to figure out what might be ailing them. The Medline website was originally intended for physicians and researchers. Now, more than a third of its visitors come from the general public.
And Medline has responded by adding twelve consumer health journals and MedlinePlus, a sister site specifically for patients. The Internet is thus not just changing the mechanism by which information is disseminated to physicians, it is changing the technology of influence, the mechanisms by which patients can affect what their physicians do.
Technology can be critical for Super Crunching to change real-world decisions. When a decision maker in business or government commissions a study, the mechanism for transmitting the results is usually not an issue. In that case, the “technology” might be as direct as handing a copy of the results to your boss. But when there are hundreds and even thousands of unsolicited studies on the general topic of interest, the question of how to quickly retrieve the most relevant result often will determine whether the result will even have a chance of altering a decision.
The Future Is Now
The success of evidence-based medicine is the rise of data-based decision making par excellence. It is decision making based not on intuition or personal experience, but on systematic statistical studies. It is Super Crunching that reversed conventional wisdom and found that beta blockers can actually help patients with heart failure. It is Super Crunching that showed that estrogen therapy does not help aging women. And it is Super Crunching that led to the 100,000 Lives Campaign.
So far, the rise of data-based decision making in medicine has largely been limited to questions of treatment. The next wave will almost certainly concern diagnosis.
The database of information that we call the Internet is already having a bizarre impact on diagnosis. The New England Journal of Medicine published a description of rounds at a New York teaching hospital. A “fellow in allergy and immunology presented the case of an infant with diarrhea; an unusual rash (‘alligator skin’); multiple immunologic abnormalities, including low T-cell function; tissue eosinophilia (of the gastric mucosa) as well as peripheral eosinophilia; and an apparent X-linked genetic pattern (several male relatives died in infancy).” The attending physicians and house staff, after a long discussion, couldn’t reach any consensus as to the correct diagnosis. Finally, the professor asked the fellow if she had made a diagnosis, and she reported that she had indeed and mentioned a rare syndrome known as IPEX which fit the symptoms perfectly. When the fellow was asked to explain how she arrived at her diagnosis, she answered: “I entered the salient features into Google, and it popped right up.” The attending physician was flabbergasted. “William Osler must be turning over in his grave. You googled the diagnosis?…Are we physicians no longer needed?”
The attending’s extemporaneous reference to William Osler is particularly apt. Osler, who was one of the founders of Johns Hopkins, is the father of the medical residency program—the continuing cornerstone of all clinical training. Osler would be turning in his grave at the thought of Google diagnoses and Google treatments because the Internet disrupts the dependence of young doctors on the teaching staff as the dominant source of wisdom. Young doctors don’t need to defer to the sage experience of their superiors. They can use sources that won’t take joy in harassing them.
A bunch of medical schools and private corporations are developing the first generation of “diagnostic-decision support” software. A diagnostic program named “Isabel” allows physicians to enter a patient’s symptoms and receive a list of the most likely causes. It will even tell the doctor whether a patient’s symptoms might be caused by the use of over 4,000 drugs. The Isabel database associates more than 11,000 specific illnesses with a host of clinical findings, lab results, patient histories, and the symptoms themselves. The Isabel programmers created a taxonomy for all of the diseases and then tutored a database by statistically searching for word patterns in journal articles that were most likely to be associated with each disease. This statistical search algorithm dramatically increased the efficiency of coding particular disease/symptom associations. And it also allows the database to continually update as new articles emerge having a high prediction of relevance. Instead of all-or-nothing Boolean searches, Super Crunching predictions about relevance are crucial to Isabel’s success.
The Isabel program grew out of a stockbroker’s own experience with suffering caused by misdiagnosis. In 1999, Jason Maude’s three-year-old daughter Isabel was misdiagnosed by a London resident as having chicken pox and sent home. It was only the next day when her organs began shutting down that Joseph Britto, an attending intensive care doctor at the hospital, realized that she in fact had a potentially fatal flesh-eating virus. Though Isabel ultimately recovered, her father was so shaken by the experience that he quit his finance job. Maude and Britto together founded a company and started to develop the Isabel software to fight misdiagnosis.
Misdiagnos
is accounts for about one-third of all medical error. Autopsy studies show that doctors seriously misdiagnose fatal illnesses about 20 percent of the time. “If you look at settled malpractice claims,” Britto said, “diagnosis error is about twice or three times as common as prescription error.” The bottom line is that millions of patients are being treated for the wrong disease. And even more troubling, a 2005 editorial in the Journal of the American Medical Association concludes that there hasn’t been a perceptible improvement in the misdiagnosis rate in the last several decades.
The ambition of Isabel is to change the stagnation in the science of diagnosis. Maude puts it simply: “Computers are better at remembering things than we are.” There are more than 11,000 diseases in the world and the human brain is not adept at remembering all the symptoms that give rise to each. Isabel actually markets itself as the Google of medical diagnosis. Like Google, it aids us in searching for and retrieving information from a large database.
The biggest reason for misdiagnosis is “premature closure.” Doctors think they have a bead on the correct diagnosis—like the resident’s idea that Isabel Maude had chicken pox—and they close their minds to other possibilities. Isabel is a reminder system of other possibilities. It actually produces a page that asks, “Have you considered…?” Just proactively reminding doctors of other possibilities can have profound effects.
In 2003, a four-year-old boy from rural Georgia was admitted to a children’s hospital in Atlanta. The boy had been sick for months, with a fever that just would not go away. The doctors on duty that day ordered blood tests, which showed that the boy had leukemia. They ordered a strong course of chemotherapy to start the very next day.