Similarly, we need to look at more and more patient data to understand the interaction of cancer with other diseases. What we do to treat a tumor may be different if a patient also has heart disease, or diabetes, or a neurodegenerative illness, and the lasting effects of treatment on those other conditions is also poorly understood right now. What it means to be sick with a certain condition, like cancer, may mean different things depending on what other comorbidities a patient faces—and all of this is a mystery right now, at least in terms of having some standard protocol and some standard set of expectations around what a patient's course will look like.
Dr. Lee compares the state of where we are right now to scientists in the sixteenth century thinking about the orbits of planets, based on discussions he's had in the past with Dr. Larry Norton of Memorial Sloan Kettering Cancer Center in New York City. As Dr. Norton has shared with Dr. Lee, there was documented data—all kinds of it—but it never really matched the motion of the planets perfectly until Johannes Kepler put it all together and realized that the orbits weren't spherical, they were elliptical—and then suddenly it all made sense. The data just needed to be looked at from a slightly different angle. That's where we are now. We are continuing to amass the data, but we haven't yet had the mindset shift that we need. And it won't come from an artificial intelligence system, but from humans looking at it differently, figuring out what they can see that no one else has seen, what we don't know yet that can explain things that we still have so much trouble explaining. The steam tables exist—we simply haven't figured them out yet.
So how do we figure out our patient equations? How do we start to move from this magic—this big black box of mysteries about disease and disease progression and the best treatments for the right patients at the right time—to a world that we can understand as easily as we can understand how to turn water into steam?
The answer, as Dr. Lee suggests, is data—but good data, matching data, useful data as the starting point for turning patient experiences into knowledge, for turning anecdotes and ideas into provable reality, into the spaces in our steam table that are just question marks right now. One thing Dr. Lee is doing is working with the Department of Defense and the Department of Veterans Affairs to create a learning health care framework that includes molecular, phenotypic, and real‐world data in a longitudinal fashion.11 Systems have to speak to each other. Data has to work. In a lot of ways, that's my story at Medidata, and it's what the next chapter is about.
Notes
1. Simon Makin, “The Amyloid Hypothesis on Trial,” Nature 559, no. 7715 (July 2018): S4–S7, https://doi.org/10.1038/d41586-018-05719-4.
2. Alaina Baker‐Nigh et al., “Neuronal Amyloid‐β Accumulation within Cholinergic Basal Forebrain in Ageing and Alzheimer's Disease,” Brain 138, no. 6 (March 1, 2015): 1722–37, https://doi.org/10.1093/brain/awv024.
3. Sahil Gupta et al., “Prostate Cancer: How Young Is Too Young?,” Current Urology 9, no. 4 (2015): 212–215, https://doi.org/10.1159/000447143.
4. “Dementia,” World Health Organization, December 12, 2017, https://www.who.int/news-room/fact-sheets/detail/dementia.
5. “Alzheimer's Disease,” Cambridge Cognition, 2014, https://www.cambridgecognition.com/cantab/test-batteries/alzheimers-disease/.
6. Joanna Walters, “Lumosity Fined Millions for Making False Claims about Brain Health Benefits,” The Guardian, January 6, 2016, https://www.theguardian.com/technology/2016/jan/06/lumosity-fined-false-claims-brain-training-online-games-mental-health.
7. Eric Wicklund, “Mobile Health App Helps Seniors Reduce Their Risk For Dementia,” mHealthIntelligence, November 27, 2017, https://mhealthintelligence.com/news/mobile-health-app-helps-seniors-reduce-their-risk-for-dementia.
8. “CCR Cancer Moonshot Projects,” Center for Cancer Research, February 14, 2018, https://ccr.cancer.gov/research/cancer-moonshot.
9. Jerry Lee, interview for The Patient Equation, interview by Glen de Vries and Jeremy Blachman, May 6, 2019.
10. Jerry S. H. Lee et al., “From Discovery to Practice and Survivorship: Building a National Real‐World Data Learning Healthcare Framework for Military and Veteran Cancer Patients,” Clinical Pharmacology & Therapeutics 106, no. 1 (April 29, 2019): 52–57, https://doi.org/10.1002/cpt.1425.
11. Ibid.
10
Good Data
In building models that work, good data management and system infrastructure are the costs of entry; there is just no way around it. Machine learning and artificial intelligence are nice buzzwords, but you need humans to set the systems up, write the rules, figure out what kinds of data need to be collected in the first place, and work to get it all “fit for purpose” from an analytic point of view. Systems have to speak to each other—in a way that produces useful output and not merely garbage. There are so many examples of bad data out there, and lessons to learn—from the Mars Lander and from the failure of IBM Watson in the oncology area, to name two well‐publicized disasters.
But even though those situations tell us what can go wrong, that knowledge doesn't, on its own, get us to something that will go right. I've been working in this area for my entire adult life, and I know data. I know what happens when it's not aggregated, standardized, and analyzed the way it needs to be to produce useful insights—and I know its power when the pieces are all properly in place.
The Failure of Watson
IBM's Watson supercomputer was supposed to revolutionize cancer treatment. A 2013 article in Wired announced, “IBM's Watson is better at diagnosing cancer than human doctors,” that it had “a breadth of knowledge no human doctor can match,” and that it had a “successful diagnosis rate for lung cancer [of] 90 percent, compared to 50 percent for human doctors.”1
Five years later, the headlines were very different. Watson was recommending “unsafe and incorrect” cancer treatments, according to a report by STAT.2 According to Becker's Hospital Review's reporting on the STAT investigation, internal documents from IBM admitted that Watson was trained on “just a few hypothetical cancer cases instead of real patient data,” was making recommendations that “conflicted with national treatment guidelines and that physicians did not find useful,” and that the product was “very limited.”3
A year after that report, IBM claims to be making “progress,”4 but the system is more like a “librarian”5 than a clinician, and simply hasn't shown the capacity for drawing novel conclusions that was promised initially. It is, as all artificial intelligence has thus far shown itself to be, only as good as the data originally entered into the system.
The Mars Climate Orbiter
On December 11, 1998, NASA launched the Mars Climate Orbiter, a $125 million robotic space probe intended to study the Martian atmosphere. Less than a year later, on September 23, 1999, communication with the orbiter was lost just as it made its way into Mars's atmosphere. It flew closer to the planet's surface than intended and was destroyed—an accident caused entirely by a mistake in the data.6 One piece of software was calculating in English units—pounds of force—while another piece of software was analyzing that output assuming it was in metric units—newtons.7 The result? The orbiter flew drastically off course and was torn apart by atmospheric friction. A “math error,” announced the Los Angeles Times headline a week after the incident—but it wasn't exactly math, it was a $125 million example of what happens when two data sets aren't able to communicate with each other.8
At Medidata, we see this all the time. We look at client data from clinical trials and sometimes there are points that are such extreme outliers that we know something is wrong. Take a height‐weight plot, for instance. Enter one person's data in centimeters instead of inches, grams instead of pounds, and suddenly he looks like the largest patient in the world. It's the most basic kind of data collection, but it's absolutely critical. As we dose some medications based on an individual's size, it can be downright dangerous if no one catches the error.
And it's not just numbers we're talking abou
t. We're capturing complex data structures like medical images as well. Imagine the sheer complexity of a single MRI. Each scan is composed of dozens of “slices” (cross‐sections of the body), each slice actually the combination and assembly of multiple images. If you've ever taken a high ISO (image sensitivity) photograph with a digital single‐lens reflex camera, you are probably familiar with the problem of noise. Static can get in the way of a clear, crisp image. The multiple images that are combined to form each MRI slice are like long‐exposure photographs with that same digital camera—they reduce noise.
Of course, digital SLRs are rapidly being replaced by the cameras in our phones, with greater and greater image quality, and increasingly less noise getting in the way. Our phones do it with multiple cameras and computational photography techniques that combine the images from all of them into one. In many ways, it is just like an MRI: all that complexity for a single slice. Then, we can assemble multiple slices into three‐dimensional images, or examine them to determine the size of a tumor. Only after all of this image processing does the data from an MRI become useful for diagnosis.
But it's not even useful enough for perfect diagnosis on its own. The MRI is only one test at one point in time that needs to be combined with all of the other data for a particular patient. We need systems that can ingest all of this data, the lab values, the images, the velocity vectors, the DNA sequences, proteomics, the ways we use our phones to check our calendars—all of the varied sources of patient equation inputs that we've talked about already—and manage the analysis, harnessing the bits of information that will help determine where we are in the phase diagrams of health. As we look at more and more complex therapies, and more complex forms of data, we need to take all the raw bits and bytes and turn them into useful information. Data can't analyze itself. The information needs to be demystified, standardized, linked together, and have the right algorithms applied to it. The job of data management done well makes the data complete, clean, and fit for the purposes of what we want that data to help us do.
The Progression to Value
No matter how sophisticated the systems used to model our health and guide our medical care are now or will be in the future, they will at their core be based on statistics and computer science. These are two disciplines where the principle of garbage in/garbage out clearly applies.
We can think of the progression toward the beneficial application of data to a specific problem as a series of steps. David Lee, Medidata's chief data officer, was the first person to teach me about the importance of data standardization and a step‐wise approach to creating value from that data. There is an absolutely critical chain of events that needs to unfold before analyses can generate true benefit. Capturing the data is obviously the first step. But disciplined data management that follows the raw capture is critical—and nontrivial—if you want to eventually derive an actual benefit from the information. Figure 10.1 shows graphically how David originally explained it to me.
The captured data must first be cleaned. We need to make sure that units are consistent and that whatever capture and integration systems we have employed (keeping in mind the potential errors from human processes, disconnected systems, and imperfect human programming) have resulted in a set of data we can feel confident in. Let us not forget the lessons of the Mars Climate Orbiter!
Figure 10.1 Creating value from data
This problem of integrating and cleaning research data in a medically and scientifically rigorous way, incidentally, is what led to the founding of Medidata in the 1990s. Working at Columbia, I saw how difficult it was to both capture and integrate data. I was certainly one of the fallible humans in the system, though I was trying my best. I remember in one case, there were lab and pathology data available in a particular hospital system, but we didn't have a terminal for it in the research building where I worked. My “data capture” process involved two elevators and crossing a street. And a handwritten laboratory notebook, of course.
This may sound like a hyperbolic anecdote from a quarter century ago, but it is still true that there are disconnected systems in our lives, and data sets that don't share a common vocabulary. It is a problem that people like Jerry Lee and so many others are still trying to solve today.
Back then, the deficiencies were even clearer. In 1994, I was sharing a lab bench with Ed Ikeguchi, a resident in the urology department. We became fast friends, sharing common interests in computer science, inventing and making things, and exploring the emerging world of the Internet. If you have ever worked in a laboratory, or even taken a lab class as part of a chemistry or biology course, you are undoubtedly familiar with how much time you can end up spending with your benchmate.
As we worked on our respective research, spying each other through a wall of test tubes and reagents, we had plenty of time to talk. Inevitably we would find ourselves lamenting the systems used for research, and how much human work was involved in transcribing and transporting data. At the time, virtually every clinical trial in the world used paper—just like my adventure crossing the street to get data and bring it back to my lab. But this was happening on a much larger scale. Data would be copied from medical charts onto paper forms, and those forms would be carried by hand, or sent—sometimes by mail, or in the most “modern” trials in the 1990s, by facsimile machines. It would then be typed into databases—not once, but twice, in an attempt to avoid yet another opportunity for transcription error.
We looked at the world of academic research, as well as industry, and the original glimmer in our eyes that would become Medidata started with a conversation about using the Internet to replace all of that work, and to facilitate far more automated data cleaning and integration. I think the question “If you can buy a book online, why can't we run trials online?” may have started it all.
It would take a few more years for us to meet Tarek Sherif and start Medidata Solutions, Inc. But in the intervening time, we began to build out some rough software ourselves, eventually creating a corporation that would end up as Medidata's precursor, and opening an “office” by moving my bed into my living room—making my one‐bedroom apartment into an incredibly small studio living space.
Thinking constantly about online research—and not sure at that point if a life at the lab bench (my original career plan) was the right thing for me—I was incredibly lucky to find an opportunity to work on an electronic medical records project emerging in our department. It was right at the nexus of my interest in computers and my interest in research, and it was there that I learned valuable lessons around the standardization and benchmarking of data—or, more honestly, about the lack of value generated when data is not standardized, making benchmarking impossible.
Carl Olsson, a great mentor and the chair of our department—along with strong support from his team of admirably forward‐looking physicians and scientists—wanted to install an electronic medical records system that would allow us to perform research projects much more quickly and effectively than the current system allowed. The grand idea was to connect research data across all of the physicians working in the department.
Unfortunately, the project was a failure. There were issues, as there often are, with technology not working as well as we had hoped, and—albeit at a relatively early point in my career—I take my share of responsibility with regard to how we managed through it. However, the greatest problem we had was with how the various members of the department wanted to record their data. Replacing paper medical charts and dictated notes—with their inherent lack of structure—was an incredibly difficult challenge to overcome. Even technologies today, like Natural Language Processing (NLP), that are meant to find structure in unstructured data (and chart notes are often cited as a perfect use case for them) can't solve some of the problems we saw back then.
Different physicians—in some cases working on the same disease, in the same building, even in the same office—had very different mental models for measuring disease progression in the
ir patients. In theory, we could have built a common data model, but the practical reality was that doctors weren't going to reinvent their thought processes in order to use it.
Compounding the problem was the fact that most electronic health records software systems—and this still applies even today—aren't designed for research, but for the business of practice management and billing. In hindsight, our inability to create that research core was inevitable. Allowing people to record data as they saw fit and were most naturally inclined to do, without common data models and processes, meant that we couldn't easily compare the data from one patient to another. We failed at data standardization. And therefore we couldn't launch into those imagined comparisons across patients and across practices. We couldn't use benchmarks. We couldn't scale research.
As we look back at the progression toward value in Figure 10.1, it really is the case that only after we are satisfied that we have high‐quality data, standardized and benchmarked, that we can unlock the higher levels of value creation. Then, finally, we can take that—the “core” we tried to achieve at Columbia—and begin to not just perform more traditional statistical analyses at scale, but also to leverage machine learning and predictive modeling to create actual benefit.
The Patient Equation Page 15