Mind in Motion

by Barbara Tversky

  Horizontal is neutral and can follow reading order. Our next question was the direction of the line. When you do research, you never know what will happen, and we were surprised by what we found. Only time corresponded to reading/writing direction. English speakers tended to map time left-to-right and Arabic speakers mapped time right-to-left, a finding that has been replicated by others. Hebrew speakers split. Could be because numbers are taught left-to-right in Hebrew-speaking schools but right-to-left in Arabic-speaking schools; could be because of greater exposure to Western languages. Now the surprise. Reading and writing direction did not influence direction of ordering quantity or preference. Nor did graphing conventions; people’s arrangements were their own inventions. In all cultures and at all ages, increases were mapped upward or leftward or rightward, but never downward, so the association of up with more was at play, but neither reading order nor graphing conventions were.

  These results aren’t as neat and tidy as we’d like, but adding them to the considerable evidence from language provides evidence that some graphing “conventions” are not arbitrary but are rooted in the ways that people think: time, a neutral dimension running horizontal, more, a value-laden dimension running vertical and upward, against gravity. These practices can conflict and can be overridden. Economists graph unemployment and inflation, both undesirable, upward, hopefully not because economists are perverse but presumably because the numbers go up, and numbers take precedence.

  That cultural artifact, reading/writing direction, turns out to have far-reaching consequences on cognition. Years ago, I saw a lovely exhibit of court paintings from northwestern India in the British Museum. Many of the paintings depicted a bevy of attractive women following—actually, chasing—an elegant self-satisfied Maharajah. In the early set of paintings, the chase went leftward, but at some point, the chase switched to going rightward. The switch in the direction of the chase seem to have happened about the time that the local written language switched direction. Movement in the direction of reading order is seen as smoother and more natural; movement in the opposite direction is seen as forced and effortful. In the first version of the cover of this book, that energetic human raced leftward. Reversing the direction to rightward instantly made the action more graceful and fluent. Like Hebrew and Arabic, Japanese is read and written right to left, so that when Japanese comics, manga, are translated to Western languages, reproducing the images always presents problems. Western soccer referees are more likely to call fouls when viewing leftward action.

  Motion is not the only characteristic that is influenced by reading and writing order. Preference and agency are as well. Images that are preferred or more powerful seem to appear more often on the left in Western languages, for example, men slightly more often than women on the left. Both preference and power go hand-in-hand with language, where preferences are often listed in order, most-preferred first, and declarations about action typically begin with the actor.

  Center/periphery

  Practices like center as focal and periphery as, well, peripheral come straight from vision: we simply see things in the center of our visual field, in the fovea, more clearly and in greater detail than things in the periphery. Presumably, whatever we focus on is, at that moment, most important to our thinking. Some maybe whimsical support for the idea: when people draw their social networks, they put themselves square in the middle. There is a perhaps apocryphal story of an African ruler at the beginning of the twentieth century who wanted to be modern and who had his country surveyed. When he learned that the capital wasn’t at the center of the country, he had it moved to the center—on the map! Far easier than moving the capital itself.

  Marks in space: Glyphs

  From Diderot, we have boxes, lines, trees/networks, and tables—rows and columns. Diderot of course was not the first to use these devices but was in good company. To enrich a semantics of diagrams, we need to add icons, symbols, dots, blobs, arrows, and a few more. These have meanings, shared by a community and related to their geometric or gestalt properties. Consider the basic three: dots, lines, enclosures; they correspond to zero, one, and two dimensions. Their senses are captured by the English prepositions, at, on, and in. Those prepositions have spatial meanings that are extended to time and more: at the corner, at two o’clock, at attention, at risk. On the tennis court, on your mark (get set, go!), on time, on drugs. In the train station, in an hour, in a muddle. The meanings of many of these devices have been established in research that can only be called empirical semantics. Some of it appears below.

  There isn’t a good term for these meaningful marks, so we will adopt a term in use, glyph. Some glyphs are synonyms. Parentheses, for example, enclose and separate, like boxes. There are many forms on our keyboards; (), [], {}. Parentheses are naturally interpreted as enclosures as they are arcs of circles facing each other and surrounding the set of things they enclose. Text turns out to be full of meaningful visual and spatial devices, like parentheses and indentation. Some glyphs (like some words) are polysemous. A circle can represent an enclosure, a general one, with an unspecified and irrelevant form. But by considering only the perimeter and not the inside, a circle can represent a cycle, a process that goes round and round with no end. Lines can connect one place to another, as in a route map, or relate one idea to another, as in a network. But lines can also be boundaries, borders, lines in the sand or red lines, lines that disappear in the tide or that are crossed despite the threat, but also boundaries that cannot be crossed.

  Elements of route maps: Points, lines, blobs

  Before we go too abstract, let’s go back to the real world, continuing to establish empirical evidence for the meanings of these devices. Years ago, we caught hungry students outside their dormitory around five o’clock and asked them if they knew where a popular fast-food place was. If they did, we asked half to sketch a map and the other half to write down directions to get there. We got highly diverse sketches and directions, some lengthy and detailed, some brief and crisp. Two appear in Figure 8.13.

  Despite that delightful diversity, we wondered whether both depictions and descriptions had the same underlying structure, and they did. Both were segmented by actions, in this case, turns. The turns were typically to new paths. Exact direction and exact distance didn’t matter, even for the sketch maps. Because they were sketches, they could have been analog, they could’ve reflected distance and direction quite accurately, but they were far from it. The route directions, in sketch or in words, consisted of a start point followed by a list of actions at choice points, usually turns to a new street or path at landmarks or intersections, ending at the desired location. Taco Bell, in this case. The sketch maps were a string of lines with some context that represented the paths or streets and points or blobs that indicated the landmarks or choice points. A part of a network. And in fact, when people in another experiment were asked to sketch a map of an entire region, they sketched what looks like a network, a configuration of points and lines, places and paths.

  FIGURE 8.13. Sketch maps of two participants.

  Points

  Points don’t move, they just stand there. At attention. An intersection, a train station, a city on a map. You or trucks or trains move from one to another, on a line, but points stay where they are. A server in a network; the server stays in place, but information goes, in a line, from server to server. Points represent anything that can be regarded as stationary, an idea in a concept map, a person in a social network. Momentary stability in a world of constant change. Lines can connect points by moving, moving from one place to another, one person to another, one thought to another.

  FIGURE 8.14. Kanizsa triangles with illusory lines.

  Lines

  Lines are everywhere, inside and out, and I am obsessed by them. There are the lines the hand draws on a page. The lines the eye sees where there are none, called Kanizsa figures after the man who demonstrated them. You can see one of his figures in Figure 8.14. The lines we create with our
bodies as we move in the world. The lines in the world, streets and buildings and bridges, the flat earth and everything parallel and perpendicular to it. Lines in the design of the world: shelves for books and toys, strings of seats in a theater, chains of buildings lining streets, rows of windows on their facades. Lines in maps and graphs.

  I am not the only one obsessed by lines. Mondrian’s Broadway Boogie Woogie, immortalizing his fascination and delight in the parallel and perpendicular, vertical and horizontal lines of Manhattan. Klee and Kandinsky, Bauhaus legends, were obsessed by these simple geometric figures, point, line, and plane. They are not simple geometric figures, they are rich concepts. Especially lines. Lines are replete with meaning and create meaning. They can be stretched and bent and curved and combined to create everything we might draw and an endless number of things we might imagine. Both Klee and Kandinsky were visual artists whose art itself did not move. Yet for both, their art was dynamic, moving. The lines did it. Klee: “A line is a dot that went for a walk.” And Kandinsky: “The line is, therefore, the very antithesis to the prototypic pictorial element—the point.” To create a line, you move your hand. Movement is inherent in lines, and lines can convey all kinds of movement. For both Klee and Kandinsky, movement, motion, moving was the elemental and natural state of the world. Both used drawing to explore and understand movement and to create it.

  Lines alone, upward, downward, straight or jagged, lines in groups, harmonious or dissonant, like music. All ways of moving.

  Containers: Blobs, circles, squares, bars

  Bar graphs and line graphs appear all over the place, from austere journals to the popular press, even as jokes. Sometimes their uses are puzzling. We wondered how viewers made sense of them. We reasoned that bars and lines are communicating different ideas even if displaying the same data. Lines show relationships; they say the points on a line have different values on the same underlying dimension. If so, lines should be interpreted as trends. By contrast, bars are containers; they say there are separate sets of things. If so, bars should be interpreted as discrete comparisons. We presented one of the graphs in Figure 8.15 to large groups of people and asked them to tell us what the graph was saying.

  FIGURE 8.15. Participants were asked to interpret one of these graphs.

  In some cases, the graphs were labeled: height of eight-year-olds and ten-year-olds or height of women and men. When the “data” were presented as lines, people gave us trends: there’s an increasing function from A to B, height increases with age, and the like. When the “data” were presented as bars, people gave us discrete comparisons: the Bs are greater than the As, ten-year-olds are taller than eight-year-olds. If these are the meanings, they should go the other way too. We asked another group to draw graphs for trends or for discrete comparisons. Sure enough, people drew lines for trends and bars for discrete comparisons. The graphic forms, lines or bars, were a larger factor in producing and understanding than the underlying data, continuous, like height, or discrete.

  Lines and boxes

  Next, we extended the investigation of the semantics of visual forms to exploration, inference, and discovery, important uses of displays of information. One task that parents, managers, and detectives share is keeping track of different people in different places at different times. One possibility is to arrange people in a time-by-place table. But if you are interested in tracking individuals over time and place, you might prefer a line graph for each person over the places. Our Three Ps (production, preference, performance) gave some support for this. We added another task that our participants enjoyed enormously. We gave them either line graphs or tables and asked them to generate as many inferences as they could. Counting inferences turned out to be more complicated than we’d thought (as I’ve said, research always brings surprises), but it seems that the tables generated more diverse inferences as well as more inferences than the line graphs. Tables gave us fascinating social and personality inferences that went far beyond the actual information, far more than line graphs. If two people were together in the same place at the same time, they must be pals. Those who went to the gym at night must be night owls. Those who didn’t go to the gym must be flabby. Tables constrain thinking less, but because they don’t suggest many inferences, they require more thinking from viewers. Lines biased interpretations toward temporal inferences. Now designers have a choice: Do they want to constrain what viewers infer or do they want to support many kinds of inferences but require viewers to explore more? A trade-off.

  Arrows

  From research on route maps, we have points, lines, and blobs—enclosures. One line has a special property, an arrow. It is asymmetric. It has a pointer, typically only at one end, and it is used to express an asymmetric relationship. Just as lines show paths as well as relations on the page, arrows show asymmetric paths and relations.

  Arrows have some foundation in experience. Arrows shot from bows fly in the direction they point. There are arrows in the sand, made by the erosion of water. Yet, unlike dots, lines, and boxes, arrows do not appear in antiquity or even the Enlightenment. Nevertheless, there were feet that walked and hands that pointed the way. The footsteps carved in the stone paving of Ephesus show the way to the brothel. Hands point in medieval texts. The arrows familiar to us now seem to proliferate only in the twentieth century. And how they proliferated! Artists like Klee and Bacon put arrows in their paintings. Math and chemistry have formalized uses of arrows. Arrows appear in road signs, sometimes confusingly so. In Venice, one of the hardest cities to navigate, signs at choice points meant to guide you to the major sights, like San Marco or Rialto, often have arrows pointing in both directions. One of many examples appears in Figure 8.16

  By now, arrows have accumulated numerous meanings. Even before they can read, American preschoolers correctly interpret arrows that indicate the direction of movement, up or down a ladder, even when the depictions are ambiguous with respect to direction. Similarly, they understand arrows used to indicate a temporal sequence of events, even when the depictions alone are ambiguous.

  FIGURE 8.16. Which way to go? Arrows indicating directions in Venice.

  To uncover the semantics of arrows, we ran pairs of studies, interpretation and production, as before. We began with tried-and-true diagrams we and others had used in previous research: a bike pump, a car brake, and a pulley system. We redesigned the diagrams so we had two sets, one with arrows showing the action, one without. We gave large groups of undergrads one of the six diagrams and asked them to describe in words what the diagram showed.

  The presence of arrows completely changed the meanings of the diagrams. Those who saw diagrams with arrows gave us step-by-step causal descriptions of the actions of the system. Part of one description of a diagram of a bike pump with arrows read: “When you push down on the handle of the bike pump, it forces air into the cylinder. That opens the valve so air can flow into the tube connected to the tire.” Notice the verbs, push, force, open, flow: all verbs of motion. One description of the pulley system with arrows included: “When the rope is pulled, the upper pulley moves, causing the middle pulley to move, which causes the lower pulley to move.” Those who saw diagrams without arrows described the structure of the systems using verbs like is and has. For the pulley system, one participant who viewed a diagram without arrows began: “There are three pulleys. One is attached to the ceiling.” For the bike pump, a participant who described a diagram without arrows said: “A bicycle pump is made of a cylinder and a handle with a piston attached at the bottom.” We coded the descriptions as structural or functional (action/behavior/cause) simply on the basis of the verbs.

  FIGURE 8.17. Diagrams produced from descriptions. The diagram on the left uses labels (don’t worry if you can’t read them) and was drawn by someone who had read a structural description of a car brake; the diagram on the right uses arrows and was drawn by someone who had read a functional description.

  We then ran the reverse experiment with new participants. We g
ave each one either a structural description of one of the systems or a functional description of one of the systems. Their job was to sketch a diagram of the system from the descriptions. Two examples appear in Figure 8.17.

  Sure enough, those who diagrammed functional descriptions used arrows, the sketch on the right. They did not label the parts. In contrast, those who diagrammed structural descriptions did not use arrows and labeled the parts, the sketch on the left. In short, arrows are understood to indicate a sequence of causal actions and are produced to diagram the same. Production and understanding (performance) mirror each other. The semantics of arrows, just like that of lines and bars, goes both ways.

  While we were working on arrows and animations, Rachel MacKenzie, an industrious undergrad, collected hundreds of diagrams appearing in STEM textbooks—biology, chemistry, physics, engineering. Although there have been claims of more than a hundred meanings to arrows, our large survey turned up around seven: to connect, for example, labels to parts; to show the next step in time; to show the next step in causation; to show motion; to show kind or direction of motion (e.g., wavy arrows); to show increases or decreases; to show invisible forces, like wind or gravity. In many cases the different uses weren’t clarified and often diagrams used arrows with three or four different senses in the same figure. You simply couldn’t know if an arrow in a diagram of the rock cycle or the nitrogen cycle showed movement or the next temporal step or an invisible force. Imagine how difficult these ambiguities are for students!