Info We Trust

by R J Andrews

  In 1905 Max Lorenz compared the cumulative share of income earned to the cumulative share of people, from poorest to richest. The Lorenz Curve highlights the deviation from the 45° line of equality.

  We have seen how to alter the position of points, but we could go further. Do you remember how large values can get visually buried when encoded in the area of shapes? We could also change how data drives other properties, such as shape area, line weight, and color value, using the same ladder of transformation. This is particularly useful when mapping populations as sized circles. A transformation can keep city populations large while also letting you see small towns. As you probe, transform to reveal more.

  And chaos theory teaches us that straight linearity, which we have come to take for granted in everything from physics to fiction, simply does not exist. Linearity is an artificial way of viewing the world. Real life isn't a series of interconnected events occurring one after another like beads strung on a necklace. Life is actually a series of encounters in which one event may change those that follow in a wholly unpredictable, even devastating way.

  IAN MALCOLM, JURASSIC PARK

  We have warmed up to data exploration with only a handful of data points at a time, across only one or two dimensions. These examples spared the room necessary to introduce, but your exploration may not be so straightforward. Many variables create complexity that is difficult to untangle. We naturally experience a multidimensional world, comprised of our many senses and a maze of memories. Yet, we are limited to only two spatial dimensions, the horizontal and vertical, to explore our data. The third dimension, depth, does not exist on paper, and is practically missing in the virtual plane of digital screens. The fourth dimension, time, is an unreliable vessel for analysis because our experience of time is so subjective. We are stuck exploring many dimensions in Flatland. What can we do?

  Climate is as reasonably a complex arena as any to take a first step into combining, expanding, and getting creative with our data sketching. It can help us add one more favorite technique on top of what we have covered so far. The scatter plot matrix is a marvelous way of scanning many pairwise patterns and comparisons across the constraints of the 2-D page, all at once, in pursuit of identifying where to dig deeper.

  Our brains, wired to detect patterns, are always looking for a signal, when instead we should appreciate how noisy the data is.

  NATE SILVER, 2012

  When we explore data, we can imagine ourselves translating and reordering like bulldozers pushing piles of rock around a quarry. Or, we are warping axis scales like time travelers morphing the fabric of space-time. You can explore beyond the techniques we have reviewed to encounter even more computational and creative approaches. As you voyage through the worlds of your data, you will go beyond these shores, learn more, and invent new ways of finding interesting patterns and comparisons.

  Hallucination was coined by Robert Browne in his 1646 Pseudodoxia Epidemica, an early inquiry into the perception of truth: “For if vision be … depraved or receive its objects erroneously, Hallucination.”

  Whatever the metaphor, interrogating data can be hard. Wrangling data can be a messy, iterative, draining ordeal. The energy required can tax the spirit. Remember, you are trying to bring some new insight into the world by mining chaos. It is difficult because it is a worthwhile endeavor.

  Across the last two chapters, we considered how to investigate with unbridled enthusiasm. In the words of Italian writer Italo Calvino, we have flown to take a look at the world from different perspectives. I hope your data sketches are rich with wondrous comparisons and patterns. Your visions have charged you to go forward with a trove of insights. But before you fully return, you might ask, was any of it real? Maybe you were just seeing things.

  Our brain is wired to see patterns even when they are not there. Critical suspicion helps protect us against becoming victim to our own statistical hallucinations. Just because a pattern appears does not mean we should believe in it.

  …certain of the evidence of my perceptions; overwhelmed by the intensity of my experience

  KARL POPPER, 1935

  Your trust in the data is the bedrock of any belief you can have in your data sketches. Understand the road the data traveled to get to your workbench. Where did it come from? How did it get here? Who touched it? Then, temper visual conclusions with a vigilant skepticism of what you think you see. But after all that, how exactly are we to evaluate the veracity of all these comparisons and patterns?

  CHAPTER

  10

  UNCERTAIN HONESTY

  Doubt is not a pleasant condition, but certainty is an absurd one.

  VOLTAIRE, 1770

  Certain. Probable. Confident. Reliable. Meaningful. Significant. These are some of the words we use to talk about how trustworthy a picture's messages are. Each one of these qualifiers is an abstract and non-visual concept. Sometimes, these words are coupled with numeric precision. They are often left naked. You do not see these qualifiers directly in the real world. For us, they are non-visual checks against our visual perception.

  The desire for truth so prominent in the quest of science, a reaching out of the spirit from its isolation to something beyond, a response to beauty in nature and art, an Inner Light of conviction and guidance—are these as much a part of our being as our sensitivity to sense impressions?

  ARTHUR EDDINGTON, 1929

  Philosopher of science Karl Popper expressed that “observation can give us ‘knowledge concerning facts' but does not justify or establish truth.” Qualifiers, like significance and confidence, characterize how truthful observations are. They are defined by abstract narratives that fill pages with verbal arithmetic. Cultural norms litter their history. Recall the model of time that Ilya Prigogine inspired; time is a stage where every instant calculates reality. The calculator takes inputs from that last moment in order to spit out the next. In a past world, certain measurements were frozen as recorded data and delivered forward to us. In the current moment, we can use data to understand the past time it came from, and use it to make guesses about what future might arrive next.

  I say not that it is, but that it seems to be.…

  HUBERT N. ALYEA, 1903–1996

  We stand today like the ancient Roman deity Janus, the god of gates, transitions, time, beginnings, and endings. Janus is depicted as having two faces, one to look into the past and one to look into the future. We do not have the luxury of perfect knowledge of other times. But we can use data to build better relationships with the past and with the future.

  Janus is sometimes shown with young and old faces to represent straddling time. Or Janus is represented with male and female faces to represent straddling dualities. The gateway between the old and new year, January, honors the god.

  Why is it so difficult for the mind's eye to see truthfulness? Let us pursue this question by looking at several statistical concepts used to improve our relationship with the truth. Together, we can develop a careful appreciation for how to qualify visual discoveries. Then, together, we can strive to picture what is so hard to see.

  A true story: I, too, have turned to lying—a much more honest lying than all the others.

  LUCIAN, c. 125–180

  Probable Possibilities

  Prediction is our first step into the world of truthfulness. Conversations about the likelihood of future events often rely on fuzzy, imprecise language. The candidate will likely win on Tuesday. Expressions such as highly unlikely, probably, and almost certainly nudge our understanding. It is an understanding with a crude specificity. Unqualified expressions often only give us a binary, more-or-less insight into what is going on. Mostly, they just tell us whether the speaker thinks one way or the other.

  CIA analyst Sherman Kent observed that Cold War policymakers interpreting the statement “serious possibility” would assign different chance values to the probability, from 20 to 80 percent.

  Subjective language expresses a psychological feeling of belief
or doubt. It does not have much scientific value. Subjective language is fuzzy, and fuzzy things are hard to refute. A word like improbable could describe a chance of 15 percent or 35 percent. A one-in-six chance could be doubtful, unlikely, or have little chance of occurring.

  Imprecise expressions reveal how we think about probability. They tell us that the context for a prediction matters a whole lot. 20 is high if you were expecting 5. But 20 is low if you were expecting 50. We often talk about probability relative to some kind of unspoken expectation. Speakers use fuzzy language because they do not have any precise numerical insight. Listeners, in turn, may not even know how to understand a specific number if they heard it. Together, they form a cabal of double-imprecision. But if a specific value is not known, then fuzzy language might be more honest than some alternatives. An analyst who conveys an uncertain result with too many decimal points invites more trust than what is warranted.

  Part of our knowledge we obtain direct; and part by argument. The Theory of Probability is concerned with that part which we obtain by argument, and it treats of the different degrees in which the results so obtained are conclusive or inconclusive. … in the actual exercise of reason we do not wait on certainty… All propositions are true or false, but the knowledge we have of them depends on our circumstances.

  JOHN MAYNARD KEYNES, 1921

  A prediction with a precise value is obtained by considering how many cases of interest there are, compared to how many equally likely cases there could be. This probability, a numerical prediction, refers to a group of possible events, not a single instance. The probability of rolling a two on a six-sided die is 1/6. This statement describes the expected reality of an entire sequence of rolls. Eventually, the total quantity of rolled twos, divided by the total number of all rolls, will stabilize at about 1/6. The next roll of the die is merely one member of a series of possible events.

  Frequentism defines an event's probability as the limit of its relative frequency in a large number of trials.

  Prediction is a prospective measure. The set of six possible die rolls is a group of fictions, only one of them will actually appear next. Prediction stands us at the threshold of the multiverse, unsure which door will pull us forward. When we evaluate a prediction, we are asked to appreciate a group of scenarios, even though we only walk one path. But prediction's possible scenarios, the multiverse of possibility, are not always so obvious as the die's six options.

  Probability's denominator, the total number of equally possible cases, asserts a claim about the chance of each possible result: They appear to us to be equally possible. Chance's randomness suggests the chaos of physics: an environment that produces largely different results because of sensitivity to small changes in initial conditions.

  The die's probability is a result of the physical symmetry of the cube. Sometimes, the set of potential outcomes is fabricated by a possibility factory. Whether we are forecasting election results or hurricane paths, statistical models work in a similar way. They simulate many possible futures, each simulation the result of slightly different initial parameters. The outcomes of these different futures can then be tallied. For example, across 1,000 simulated hurricane paths, 341 show the storm hitting the coast and 659 do not, yielding a 34 percent chance of landfall occurring.

  Chance is a reflection on our ability to measure the system. A die throw can actually be predicted by an experimenter who knows the die's initial position and has a high-speed camera, knowledge of the surface it is rolled on, and a three-dimensional model of how the die tumbles. This effort is impractical for the gamer, and so we can consider the die roll random. The chance phenomenon in reality is as much a reflection on our own perception as it is the events observed.

  Like a toy train, models are incomplete simplifications of reality. They rely on assumptions. Models also harbor the biases of their human tenders. They do not fully capture the richness of the world. But, neither do our minds! Like our own understandings, models are approximations, hopefully good-enough for the task. And models can get better over time as they are fed more data. But here is the visual challenge: models are popularly thought of as black boxes. The dominant way of describing them is an object which you cannot peer into. At their invisible worst, a statistical model is proprietary. But even the mechanisms of a published model are obtuse unless you work in its field. Models often require some kind of faith that, behind the curtain, a fair multiverse of possibilities is being created. The model asks you to trust that its input data, assumptions, and rules are all designed well enough.

  Stare at the world, not at your model.

  ARNOLD KLING, 2017

  Eventually, one of the possible futures of the multiverse actually manifests. But we, the audience, may still not know much more about the validity of the model. Imagine a binary choice between success and failure. Model A predicts a 60 percent chance of success. Model B predicts a 90 percent chance of success. If the event is a success, was model A or B more correct? We cannot know with only a single event. The event would have to be run again and again to determine if the success rate were closer to 60 or 90 percent.

  Just as the ability to devise simple but evocative models is the signature of the great scientist so overelaboration and overparameterization is often the mark of mediocrity. Since all models are wrong the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.

  GEORGE E.P. BOX, 1976

  The way we interpret numerical probability is also fuzzy. It is too natural for us to translate a number to a feeling. If you read that “a team's chance of winning the game is 83 percent,” you might translate this to a subjective statement such as my team is probably going to win. You might even believe your team is certain to win. A more fair digestion of 83 percent might be: If the same game happened six times, my team would lose once.

  If a meteorologist says there is a 70% chance of rain and it doesn't rain, is she wrong? Not necessarily. Implicitly, her forecast also says there is a 30% chance it will not rain.… [people] always judge the same way: they look at which side of “maybe”—50%—the probability was on. If the forecast said there was a 70% chance of rain it rains, people think the forecast was right; if it doesn't rain, they think it was wrong.

  TETLOCK AND GARDNER, 2015

  Consider a candidate with a 44 percent chance of winning an election. The first way you might see this number is some kind of progress bar, almost halfway to 100. We obsess over the halfway mark. The chance of winning is confused with the 50 percent share of the vote needed to win the election. The vision of the progress bar arrests our appreciation of what this probability really means. Instead, imagine blindly throwing a dart at a board that is 44 percent covered in blue for the candidate. Now, fine differences in probability, and proximity to 50 percent, do not seem to matter as much. So, when are these specific numbers any use? No surprise, we engage once we are able to compare.

  We can only reasonably predict so far into the future. The Lyapunov Time helps define the prediction horizon, how far into the future we can hope to make predictions for a system: millions of years for a planet's orbit; a few days for the weather.

  Now, consider two candidates chances of winning: 44 versus 43 percent. Suddenly the contrast gives context, a comparison for us to bite into. Numbers become much more interesting if we can focus on their difference. Similarly, a change in prediction for a single candidate—40 percent last week, 44 percent this week— provides an easy narrative: The candidate is trending up. A single number lacks precise meaning without context. Two predictions create an environment where precision suddenly seems to matter a great deal. But we should not forget that without the comparison, specific numbers are lost on us.

  Measurements are only useful for comparison. The context supplies a basis for the comparison— perhaps a baseline, a benchmark, or a set of measures for intercomparison. Sometimes the baseline is implicit, based on general knowledge, as when
the day's temperature is reported and can be related to local knowledge and past experience.

  STEPHEN STIGLER, 2016

  Elections send us forward with a question about the mental pitfalls of reading statistical claims. How do we factor in trust? Is there any use in reporting a one-point difference between candidates if the margin of error is three points? Comparison without context about certainty creates a false sense of confidence.

  Percentiles, the multiverse of probability, black-box models, and our subjective reading of it all combine for quite a quagmire. The language of truthfulness gets mixed up with the language of belief. What exactly constitutes confidence? At what point is something significant? Is the margin of error significant? How exactly do we measure trust?

  “We very soon got to six yards to the mile. Then we tried a hundred yards to the mile. And then came the grandest idea of all! We actually made a map of the country, on the scale of a mile to the mile!” “Have you used it much?” I enquired. “It has never been spread out, yet,” said Mein Herr: “the farmers objected: they said it would cover the whole country, and shut out the sunlight ! So we now use the country itself, as its own map, and I assure you it does nearly as well.”