Understanding Context

by Andrew Hinton

It then works with abstract representations of what is sensed.

  The brain processes the representational data by using disembodied rules, models, and logical operations.

  The brain then takes this disembodied, abstract information and translates it into instructions to the body.

  In other words, according to the mainstream view, perception is indirect, and requires representations that are processed the way a computer would process math and logic. The model holds that this is how cognition works for even the most basic bodily action.

  This computer-like way of understanding cognition emerged for a reason: modern cognitive science was coming of age just as information theory and computer science were emerging as well; in fact, the “cognitive revolution” that moved psychology past its earlier behaviorist orthodoxy was largely due to the influence of the new field of information science.[28]

  So, of course, cognitive science absorbed a lot of perspectives and metaphors from computing. The computer became not just a metaphor for understanding the brain, but a literal explanation for its function. It framed the human mind as made of information processed by a brain that works like an advanced machine.[29] This theoretical foundation is still influential today, in many branches of psychology, neuroscience, economics, and even human-computer interaction (HCI).

  To be fair, this is a simplified summary, and the disembodied-cognition perspective has evolved over time. Some versions have even adopted aspects of competing theories. Still, the core assumptions are based on brain-first cognition, arguing that at the center is a “model human processor” that computes our cognition using logical rules and representations, much like the earliest cognitive scientists and HCI theorists described.[30] And let’s face it, this is how most of us learned how the brain and body function; the brain-is-like-a-computer meme has fully saturated our culture to the point at which it’s hard to imagine any other way of understanding cognition.

  The mainstream view has been challenged for quite a while by alternative theories, which include examples such as activity theory, situated action theory, and distributed cognition theory.[31] These and others are all worth learning about, and they all bring some needed rigor to design practice. They also illustrate how there isn’t necessarily a single accepted way to understand cognition, users, or products. For our purposes, we will be exploring context through the perspective of embodied cognition theory.

  Embodied Cognition: An Alternative View

  In my own experience, and in the process of investigating this book’s subject, I’ve found the theory of embodied cognition to be a convincing approach that explains many of the mysteries I’ve encountered over my years of observing users and designing for them.

  Embodied cognition has been gaining traction in the last decade or so, sparking a paradigm shift in cognitive science, but it still isn’t mainstream. That’s partly because the argument implies mainstream cognitive science has been largely wrong for a generation or more. Yet, embodied cognition is an increasingly influential perspective in the user-experience design fields, and stands to fundamentally change the way we think about and design human-computer interaction.[32]

  Generally, the embodiment argument claims that our brains are not the only thing we use for thought and action; instead, our bodies are an important part of how cognition works. There are multiple versions of embodiment theory, some of which still insist the brain is where cognition starts, with the body just helping out. However, the perspective we will be following argues that cognition is truly environment-first, emerging from an active relationship between environment, body, and brain.[33] As explained by Andrew Wilson and Sabrina Golonka in their article “Embodied Cognition Is Not What You Think It Is”:

  The most common definitions [of embodied cognition] involve the straightforward claim that “states of the body modify states of the mind.” However, the implications of embodiment are actually much more radical than this. If cognition can span the brain, body, and the environment, the “states of mind” of disembodied cognitive science won’t exist to be modified. Cognition will instead be an extended system assembled from a broad array of resources. Taking embodiment seriously therefore requires both new methods and theory.[34]

  The embodied approach insists on understanding perception from the first-person perspective of the perceiving organism, not abstract principles from a third-person observer. A spider doesn’t “know” about webs, or that it’s moving vertically up a surface; it just takes action according to its nature. A bird doesn’t “know” it is flying in air; it just moves its body to get from one place to another through its native medium. For we humans, this can be confusing, because by the time we are just past infancy, we develop a dependence on language and abstraction for talking and thinking about how we perceive the world—a lens that adds a lot of conceptual information to our experience. But the perception and cognition underlying that higher-order comprehension is just about bodies and structures, not concepts. Conscious reflection on our experience happens after the perception, not before.

  How can anything behave intelligently without a brain orchestrating every action? To illustrate, let’s look at how a Venus flytrap “behaves” with no brain at all. Even though ecological psychology and embodiment are not about plants, but terrestrial animals with brains and bodies that move, the flytrap is a helpful example because it illustrates how something that seems like intelligent behavior can occur through a coupled action between environment and organism.

  The Venus flytrap (Figure 4-3) excretes a chemical that attracts insects. The movement of insects drawn to the plant then triggers tiny hairs on its surface. These hairs structurally cause the plant to close on the prey and trap it.

  Figure 4-3. A Venus flytrap—complex behavior without a brain[35]

  This behavior already has some complexity going on, but there’s more: the trap closes only if more than one hair has been triggered within about 20 seconds. This bit of conditional logic embodied by the plant’s structure prevents it from trapping things with no nutritional value. Natural selection evidently filtered out all the flytraps that made too many mistakes when catching dinner. This is complex behavior with apparent intelligence underpinning it. Yet it’s all driven by the physical coupling of the organism’s “body” and a particular environmental niche.

  Now, imagine an organism that evolved to have the equivalent of millions of Venus flytraps, simple mechanisms that engage the structures of the environment in a similar manner, each adding a unique and complementary piece to a huge cognition puzzle. Though fanciful, it is one way of thinking about how complex organisms evolved, some of them eventually developing brains.

  In animals with brains, the brain enhances and augments the body. The brain isn’t the center of the behavioral universe; rather, it’s the other way around. It’s this “other way around” perspective that Gibson continually emphasizes in his work.

  Figure 4-4 illustrates a new model for the brain-body-environment relationship.

  Figure 4-4. A model for embodied cognition

  In this model, there’s a continuous loop of action and perception in which the entire environment is involved in how a perceiver deciphers that environment, all of it working as a dynamical, perceptual system. Of course, the brain plays an important role, but it isn’t the originating source of cognition. Perception gives rise to cognition in a reciprocal relationship—a resonant coupling—between the body and its surroundings.

  This perception-action loop is the dynamo at the center of our cognition. In fact, perception makes up most of what we think of as cognition to begin with. As Louise Barrett puts it in Beyond the Brain: How Body and Environment Shape Animal and Human Minds (Princeton University Press), “Once we begin exploring the actual mechanisms that animals use to negotiate their worlds, it becomes hard to decide where ‘perception’ ends and ‘cognition’ starts.”[36] Just perceiving the environment’s information already does a lot of the work that we often attribute to brain-based cognit
ion.

  Embodiment challenges us to understand the experience of the agent not from general abstract categories, but through the lived experience of the perceiver. One reason I prefer this approach is that it aligns nicely with what user experience (UX) design is all about: including the experiential reality of the user as a primary input to design rather than relying only on the goals of a business or the needs of a technology. Embodied cognition is a way of understanding more deeply how users have experiences, and how even subtle changes in the environment can have profound impacts on those experiences.

  Using the Environment for Thinking

  Designing for digital products and services requires working with a lot of abstractions, so it’s helpful to bring those abstractions out of the cloudy dimension of pure thought and into the dimension of physical activity. This is why we so often find ourselves using our bodily environment for working out design problems and why it’s emerged as a recognized best practice.

  As an example let’s consider how we use an office stalwart: the sticky note. By using sticky notes, we can move language around on a physical surface. As we’ll see, language makes it possible for us to use bits of semantic information (labels, phrases, icons) as stand-ins for what they represent—anything from simple objects to large, complex ideas. By using the physical surface and the uncannily just-sticky-enough adhesive on the notes (Figure 4-5), we not only make use of the spatial relationships between notes to discover affinities and create structures, but also engage our bodies in thinking through the problem.

  Sketching is another way we can externalize thought into bodily engagement with our environment—whether we’re working through diagrammatic models to discover and rehearse abstract structural relationships, or we’re informally playing around with representations of actual objects and interfaces. Sketching isn’t only about what’s being put on paper or drawn on-screen; sketching also engages our bodies in working through the contours of structure and potential action. Sketching can come in many forms, from chalk on a blackboard to CAD drawings or “wireframes” to making quick-and-cheap physical prototypes.

  Figure 4-5. Using sticky notes to work through abstractions

  Figure 4-6. Kate Rutter, live-sketchnoting Karen McGrane’s closing plenary at IA Summit 2013[37]

  Action and the Perceptual System

  As shown in the perception-action loop of Figure 4-4, we understand our environment by taking action in it. Gibson stresses that perception is not a set of discrete inputs and outputs, but happens as a perceptual system that uses all the parts of the system at once, where the distinction between input and output is effaced so that they “are considered together so as to make a continuous loop.”[38] The body plays an active part in the dynamical feedback system of perception. Context, then, is also a result of action by a perceiving agent, not a separate set of facts somehow insulated from that active perception. Even when observing faraway stars, astronomers’ actions have effects on how the light that has reached the telescope is interpreted and understood.

  It’s important to stress how deeply physical action and perception are connected, even when we are perceiving “virtual” or screen-based artifacts. In the documentary Visions of Light: The Art of Cinematography, legendary cameraman William Fraker tells a story about being the cinematographer on the movie Rosemary’s Baby. At one point, he was filming a scene in which Ruth Gordon’s spry-yet-sinister character, Minnie, is talking on the phone in a bedroom. Fraker explains how director Roman Polanski asked him to move the camera so that Minnie’s face would be mostly hidden by the edge of a doorway, as shown in Figure 4-7. Fraker was puzzled by the choice, but he went along with it.

  Fraker then recounts seeing the movie’s theatrical premiere, during which he noticed the audience actually lean to the right in their seats in an attempt to peek around the bedroom door frame. It turned out that Polanski asked for the odd, occluding angle for a good reason: to engage the audience physically and heighten dramatic tension by obscuring the available visual information.

  Even though anyone in the theater would have consciously admitted there was no way to see more by shifting position, the unconscious impulse is to shift to the side to get a better look. It’s an intriguing illustration of how our bodies are active participants in understanding our environment, even in defiance of everyday logic.

  Figure 4-7. Minnie’s semi-hidden phone conversation in Rosemary’s Baby (courtesy Paramount Pictures)[39]

  Gibson uses the phrase perceptual system rather than just “the eye” because we don’t perceive anything with just one isolated sense organ.[40] Perception is a function of the whole bodily context. The eye is made of parts, and the eye itself is a part of a larger system of parts, which is itself part of some other larger system. Thus, what we see is influenced by how we move and what we touch, smell, and hear, and vice versa.

  In the specific case of watching a movie, viewers trying to see more of Minnie’s conversation were responding to a virtual experience as if it were a three-dimensional physical environment. They responded this way not because those dimensions were actually there, but because that sort of information was being mimicked on-screen, and taking action—in this case leaning to adjust the angle of viewable surfaces—is what a body does when it wants to detect richer information about the elements in view. As we will see in later chapters, this distinction between directly perceived information and interpreted, simulated-physical information is important to the way we design interfaces between people and digital elements of the environment.

  This systemic point of view is important in a broader sense of how we look at context and the products we design and build. Breaking things down into their component parts is a necessary approach to understanding complex systems and getting collaborative work done. But we have a tendency (or even a cognitive bias) toward forgetting the big picture of where the components came from. A specific part of a system might work fine and come through testing with flying colors, but it might fail once placed into the full context of the environment.

  Information Pickup

  Gibson coins the phrase information pickup to express how perception picks up, or detects, the information in the environment that our bodies use for taking action. It’s in this dynamic of information pickup that the environment specifies what our bodies can or cannot do. The information picked up is about the mutual structural compatibility for action between bodies and environments.

  In the same way a weather vane’s “body” adjusts its behavior directly based on the direction of wind, an organism’s biological structures respond to the environment directly. A weather vane (Figure 4-8) moves the way it does because its structure responds to the movement of air surrounding it. Similarly, the movements of the elbow joint also shown in Figure 4-8 are largely responses to the structure of the environment. When we reach for a fork at dinner or prop ourselves at the table, the specifics of our motion don’t need to be computed because their physical structure evolved for reaching, propping, and other similar actions. The evolutionary pressures on a species result in bodily structures and systems that fit within and resonate with the structures and systems of the environment.[41]

  Figure 4-8. Weather vanes[42] and the human elbow joint[43] respond to their environments.

  Information pickup is the process whereby the body can “orient, explore, investigate, adjust, optimize, resonate, extract, and come to an equilibrium.”[44] Most of our action is calibrated on the fly by our bodies, tuning themselves in real time to the tasks at hand.[45] When standing on a wobbly surface, we squat a bit to lower our center of gravity, and our arms shoot out to help our balance. This unthinking reaction to maintain equilibrium is behind more of our behavior than we realize, as when moviegoers leaned to the side to see more of Minnie’s phone conversation. There’s not a lot of abstract calculation driving those responses. They’re baked into the whole-body root system that makes vision possible, responding to arrays of energy interacting with the surfaces of
the environment. Gibson argued that all our senses work in a similar manner.

  Affordance

  If you’ve done much design work, you’ve probably encountered talk of affordances. The concept was invented by J.J. Gibson, and codeveloped by his wife, Eleanor. Over time, affordance became an important principle for the Gibsons’ theoretical system, tying together many elements of their theories.

  Gibson explains his coining of the term in his final, major work, The Ecological Approach to Visual Perception:

  The affordances of the environment are what it offers the animal, what it provides or furnishes, either for good or ill. The verb to afford is found in the dictionary, but the noun affordance is not. I have made it up. I mean by it something that refers to both the environment and the animal in a way that no existing term does. It implies the complementarity of the animal and the environment.[46]

  More succinctly: Affordances are properties of environmental structures that provide opportunities for action to complementary organisms.[47]

  When ambient energy—light, sound, and so on—interacts with structures, it produces perceivable information about the intrinsic properties of those structures. The “complementary” part is key: it refers to the way organisms’ physical abilities are complementary to the particular affordances in the environment. Hands can grasp branches, spider legs can traverse webs, or an animal’s eyes (and the rest of its visual system) can couple with the structure of light reflected from surfaces and objects. Affordances exist on their own in the environment, but they are partly defined by their relationship with a particular species’ physical capabilities. These environmental and bodily structures fit together because the contours of the latter evolved within the shaping mold of the former.