In Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Oxford University Press), Andy Clark argues that our bodies and brains move with great fluidity between various sorts of cognition. From moment to moment, our cognition uses various combinations of cognitive loops—subactive cognition, active-body cognition, and extended cognition, using the scaffolding of the environment around us. Clark explains that our minds “are promiscuously body-and-world exploiting. They are forever testing and exploring the possibilities for incorporating new resources and structures deep into their embodied acting and problem-solving regimes.”[61]
Clark also explains how an assembly principle is behind how cognition works as efficiently as it can, using “whatever mix of problem-solving resources will yield an acceptable result with minimal effort.”[62] We might say that we use a combination of “loops of least effort.”[63] That’s why audience members leaned to the right in a theater, even though there was no logical reason to do so. We act to perceive, based on the least effortful interpretation of the information provided, even though it sometimes leads us astray. That is, we can easily misinterpret our context, and act before our error is clear to us. Even though we might logically categorize the variety of resources the perceiver recruits for cognition, to the perceiver it is all a big mash-up of information about the environment. For designers, that means the burden is on the work of design to carefully parse how each element of an environment might influence user action, because the user will probably just act, without perceiving a difference.
“Satisficing”
Perhaps this loops-of-least-effort idea helps explain a behavior pattern first described by scientist-economist Herbert Simon, who called it satisficing. Satisficing is a concept that explains how we conserve energy by doing whatever is just enough to meet a threshold of acceptability. It’s a portmanteau combining “satisfy” and “suffice.” Its use has been expanded to explain other phenomena, from how people decide what to buy to the way a species changes in response to evolutionary pressures.[64] Satisficing is a valuable idea for design practice, because it reminds us that users use what we design. They don’t typically ponder it, analyze it, or come to know all its marvelous secrets. They act in the world based on the most obvious information available and with as little concentration as possible. That’s because cognition starts with, and depends upon, continual action and interaction with the environment. Users aren’t motivated by first understanding the environment. They’re too busy just getting things done, and in fact they tend to improvise as they go, often using the environment in different ways than intended by designers.[65]
Even the most careful users eventually “poke” the environment to see how it responds or where it will take them by clicking or tapping things, hovering with a mouse, waving a controller or phone around in the air, or entering words into a search field. Just the act of looking is a physical action that probes the environment for structural affordance information, picking up the minimum that seems to be needed to move and then appropriating the environment to their needs. We see this when we observe people using software: they’ll often try things out just to see what happens, or they find workarounds that we never imagined they would use. It does no good to call them “bad users.” These are people who behave the way people behave. This is one reason why lab-based testing can be a problem; test subjects can be primed to assume too much about the tested artifact, and they can overthink their interactions because they know they’re being observed. Out in the world they are generally less conscious of their behaviors and improvised actions.
The embodied view flips the traditional role of the designer. We’re used to thinking of design as creating an intricately engineered setting for the user, for which every act has been accounted. But the contextual meaning of the environment is never permanently established, because context is a function of the active engagement of the user. This means the primary aim of the designer is not to design ways for the artifact to be used but instead to design the artifact to be clearly understood,[66] so the user can recruit it into her full environmental experience in whatever way she needs.
Umwelts
We have a Boston terrier named Sigmund (Figure 4-10). He’s a brownish-red color, unlike typical Bostons. When walking Sigmund, I notice that no matter how well he’s staying by my side, on occasion he can’t help going off-task. Sometimes, he stops in his tracks, as if the ground has reached up and grabbed him. And, in a sense, that’s what is happening. Sigmund is perceiving something in his environment that is making him stop. It’s not premeditated or calculated; it’s a response to the environment not unlike walking into a glass wall. For him, it’s something he perceives as “in the way” or even dangerous, like an angle of shadow along the ground that could be a hole or something closing in on him. It happens less as he gets older and learns more about the world around him. Stairs used to completely freak him out, but now he’s a pro.
Figure 4-10. Sigmund
But often what affects Sigmund’s behavior is invisible to me. Like many dogs, he’s much more perceptive of sound and smell than hominids. In those dimensions, Sigmund’s world is much richer than mine. If I bring him outside and we encounter even a mild breeze, he’ll stop with his nose pointed upward and just smell the air the way I might watch a movie at IMAX. Certainly we make friends differently. At the dog park, I’m mostly paying attention to the visual aspects of faces around me, whereas Sigmund gets to know his kind from sniffing the other end.[67]
For Sigmund and me, much of our worlds overlap; we’re both warm-blooded terrestrial mammals, after all. We’re just responding to different sorts of information in addition to what we share. Sigmund might stop because of a scent I cannot smell, but I might stop in my tracks because I see a caution light or stop sign. Sigmund and I are walking in somewhat different worlds—each of us in our own umwelt.
Umwelt is an idea introduced by biologist Jakob Johann von Uexküll (1874–1944), who defined it as the world as perceived and acted upon by a given organism. Uexküll studied the sense organs and behaviors of various creatures such as insects, amoebae, and jellyfish, and developed theories on how these they experience their environments.[68]
As a result of this work, Uexküll argued that biological existence couldn’t be understood only as molecular pieces and parts, but as organisms sensing the world as part of a system of signs. In other words, Uexküll pioneered the connection of biology with semiotics, creating a field now called biosemiotics (we will look more closely at signs, signification, and semiotics in Part III). His work also strongly influenced seminal ideas in phenomenology from the likes of Martin Heidegger and Maurice Merleau-Ponty.
If we stretch Uexküll’s concept just a little, we can think of different people as being in their own umwelts, even though they are the same species. Our needs and experiences shape how we interpret the information about the structures around us.
For skateboarders, an empty swimming pool has special meaning for activities that don’t register for nonriders; for jugglers, objects of a certain size and weight, such as oranges, can mean “great for juggling,” whereas for the rest of us, they just look delicious.
When you’re looking for a parking place at the grocery store, you notice every nuance that might indicate if a space is empty. After you park, you could probably recall how many spaces you thought were empty but turned out to just have small cars or motorcycles parked in them. But, when you’re leaving the store, you’re no longer attending to that task with the same level of explicit concentration; you might notice no empty spaces at all. Instead, you’re trying to locate the lot’s exit, which itself can be an exercise in maddening frustration. The parking lot didn’t change; the physical information is the same. But your perspective shifted enough that other information about different affordances mattered more than it did earlier.
In the airport scenario in Chapter 1, when I was conversing with my coworker about my schedule, my perception was different from his b
ecause the system for understanding the world that I was inhabiting (that is, my view of the calendar) was different from his, even though we are the same species, and even fit the same user demographics.
This idea of umwelt can also help us understand how digital systems, when given agency, are a sort of species that see the world in a particular way. Our bodies are part of their environment the way their presence is part of ours. New “smart” products like intelligent thermostats and self-driving cars—and even basic websites and apps—essentially use our bodies as interfaces between our needs and their actions. When we design the environments that contain such agents, it’s valuable to ask: what umwelt are these agents living in, and how do we best translate between them and the umwelt of their human inhabitants?
* * *
[18] I should note that in James Gibson’s work, he never called this sort of information “physical information.” He was careful to use “physical” specifically for describing properties of the world that exist regardless of creaturely perception. A more accurate term might be “ecological” or “directly perceived” information, but this is an academic distinction that I found unhelpful to design audiences. So, for simplicity, I’ve opted to call the mode “physical” instead.
[19] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979: 242.
[20] ———. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979: 239.
[21] ———. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979: 255.
[22] Russ Hamilton, photographer. Cornell University Photo Services. Division of Rare and Manuscript Collections, Cornell University Library.
[23] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979: 148
[24] ———. Sensory processes and perception. A Century of Psychology as a Science. American Psychological Association, Washington, DC, 1992: 224–230. (ISBN 155798171X)
[25] James Gibson actually studied under a protege of William James, E.B. Holt. Burke, F. Thomas. What Pragmatism Was. Indiana University Press, 2013: 92.
[26] Gibson, E. J., and R. D. Walk. “The visual cliff.” Scientific American, 1960; 202:64–72.
[27] McCullough, Malcolm. [Based on a diagram in] Ambient Commons: Attention in the Age of Embodied Information. Cambridge, MA: MIT Press, 2013.
[28] Gleick, James. The Information: A History, a Theory, a Flood. New York: Random House, Inc., 2011: 4604–5, Kindle edition.
[29] Louise Barrett traces the origin of the brain-as-computer metaphor to the work of John von Neumann in the late 1940s. Barrett, Louise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:121, Kindle edition.
[30] Durso, Francis T. et al. Handbook of Applied Cognition. New York: John Wiley & Sons, 2007.
[31] Nardi, Bonnie. Context and Consciousness: Activity Theory and Human-Computer Interaction. “Studying Context: A Comparison of Activity Theory, Situated Action Models, and Distributed Cognition” pp. 35–52.
[32] Kirsh David. Embodied Cognition and the Magical Future of Interaction Design. ACM Trans. On Human Computer Interaction 2013.
[33] Including the entire environment as part of cognition has also been called “extended cognition theory.” We’ll just be using “embodied” as the umbrella term.
[34] Wilson, A., and S. Golonka. Frontiers in Psychology February 2013; Volume 4, Article 58.
[35] Wikimedia Commons: http://bit.ly/1xauZXB
[36] Barrett, Louise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:55.
[37] Photo by Jared Spool.
[38] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:245.
[39] A frame captured from the streamed version of the film, reproduced under fair use.
[40] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:244–6.
[41] Gibson, J. J., The Senses Considered as Perceptual Systems. Boston: Houghton Mifflin, 1966:319.
[42] Wikimedia Commons: http://bit.ly/1rYwHIj
[43] From Gray’s Anatomy. Wikimedia Commons: http://bit.ly/1CM8viC
[44] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:245.
[45] It bears mentioning that Gibson’s idea of “calibration” Norbert Wiener’s cybernetics concept of “feedback.” For more on Wiener, see: Bates, Marcia J. Information. In Encyclopedia of Library and Information Sciences. 3rd ed. 2010; Bates, Marcia J, Mary Niles Maack, eds. New York: CRC Press, Volume 3, pp. 2347–60. Retrieved from Scenario: Andrew Goes to the Airport–Scenario: Andrew Goes to the Airport of http://bit.ly/1wkajgO.
[46] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:127.
[47] This is a slight paraphrase from Golonka, Sabrina. “A Gibsonian analysis of linguistic information.” Posted in Notes from Two Scientific Psychologists June 24, 2014. http://bit.ly/1rYwsgm.
[48] Jones, K. “What Is An Affordance?” Ecological Psychology 2003;15(2):107–14.
[49] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:138–9.
[50] ———. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:140.
[51] Dotov, Dobromir G, Lin Nie, Matthieu M de Wit. “Understanding affordances: history and contemporary development of Gibson’s central concept.” Avant 2012;3(2).
[52] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:137.
[53] ———. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979: 21.
[54] ———. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:42.
[55] Norman does, however, explain that he is “appropriating” Gibson’s concept and using it in a different way than Gibson intended. (Norman, 2013:14). My focus on affordance here has attempted to bring more Gibson into our understanding of the theory.
[56] Golonka, Sabrina. “A Gibsonian analysis of linguistic information.” Posted in Notes from Two Scientific Psychologists June 24, 2014 (http://bit.ly/1rYwsgm).
[57] Barrett, Louise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:172.
[58] Meinhold, Bridgette. “Bandaid-Like Stick-On Circuit Board Turns Your Body Into a Gadget.” Ecouterre April 15, 2014 (http://bit.ly/1nx6TYB).
[59] Haggard, Patrick, Matthew R Longo. “You Are What You Touch: How Tool Use Changes the Brain’s Representations of the Body.” Scientific American September 7, 2010 (http://bit.ly/1FrVPll).
[60] Barrett, Louise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:199.
[61] Clark, A. Supersizing the Mind: Embodiment, Action, and Cognitive Extension London: Oxford University Press, 2010:42.
[62] ———. Supersizing the Mind: Embodiment, Action, and Cognitive Extension London: Oxford University Press, 2012:568–569, Kindle locations.
[63] “Loops of least effort” is not Clark’s phrasing, but my own. This is an idea we’ll return to later, when we see how users tend to rely on physical information before bothering with the extra effort required by most semantic information.
[64] Barrett, Louise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:128, Kindle edition.
[65] Dourish, Paul. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press, 2005:170.
[66] ———. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press, 2005:172.
[67] Interestingly, though, due to millennia of breeding, dogs depend on faces more than scent or body language for r
ecognizing and relating to humans. Gill, Victoria. “Dogs recognize their owner’s face” BBC Earth News October 22, 2010 (http://bbc.in/1nx6Snh).
[68] http://en.wikipedia.org/wiki/Umwelt
Chapter 5. Attention, Control, and Learning
Never memorize something that you can look up.
—ALBERT EINSTEIN
A Spectrum of Conscious Attention
CONTEXT IS A FUNCTION OF UNDERSTANDING THE ENVIRONMENT, which involves our consciousness. To what degree, however, are we really conscious of our environment, or even our consciousness? We like to think we’re logical, rational beings that take action mainly out of higher-order thought. Yet, as is explored in Chapter 4, our bodies do a lot of the thinking for us. It turns out that the environment is also responsible for much of our decision-making, attention, and learning. Embodied cognition is part of a general trend in the past few decades in which new schools of thought are questioning long-held assumptions that go back at least as far as Rene Descartes’ supposition that “I think, therefore I am.”
Understanding Context Page 8