Understanding Context

by Andrew Hinton

  We are a symbol-laden species, so the way we talk about the world pervades the world, affecting how we understand it and use it, regardless of how our bodies perceive intrinsic affordances. Just as a fallen branch might have an affordance of being picked up and held in the hand and then used as an extension of one’s arm, a word for the fallen branch—like “club” or “kindling”—offers a new dimension of meaning for the branch, a semantic function implying certain kinds of action. And, it does this without anyone having to actually pick up the stick and use it.

  In that sense, for language-comprehending humans, a label can alter what the object actually is. It doesn’t change the physical form of the object, but for all practical purposes in human life, it’s a different object. When we design products and services, we are designing in the “for all practical purposes” realm. In that realm, language functions as environment. That’s because semantic function is not merely abstract; it shapes our very reality.

  Tools for Understanding

  Semantic function allows us to use words to name and organize our world. Gibson saw language as something that transforms the way knowledge works; it makes “knowing explicit instead of tacit. Language permits descriptions and pools the accumulated observations of our ancestors.”[199] It turns knowledge into environment that can persist with us across time, even if only as stories imperfectly told from one generation to the next.

  Andy Clark calls language “a form of mind-transforming cognitive scaffolding: a persisting, though never stationary, symbolic edifice [playing a] critical role in promoting thought and reason.”[200] We use language to add structures to our environment, structures that inform us in ways that wouldn’t exist without language. A conversation creates environmental structures that we pick up as having meaning that supports collaborative action. A set of written instructions guides us in building a house or a piece of furniture, which we would not know how to create without those plans.

  What this means is, language is infrastructure. Language can actually create new invariants for us to interact with and inhabit. It can have the effect of creating new elements in the environment—and even new environments. Symbolic language affords fluid usage of words for the things around us. Because there’s not a necessary one-to-one, never-changing relationship between word and object, we can label things in many ways. This labeling function is one of language’s most powerful abilities. Labeling is how we bring stability to our experience and make explicit sense of the world as humans.[201]

  Clark describes this as language creating “a new realm of perceptible objects.”[202] Here are the key points he makes about labels in particular:

  Labeling “functions as a kind of augmented reality trick,” where we supplement our surroundings with new structure.

  Labels are “cheap” ways to group objects without having to actually move items around (and in some cases group things that we couldn’t physically move into piles anyway).

  Labels are open-ended in the sense that we can group things that have no physically evident affordance for doing so.

  Labels behave much like physical tools for piling (and, for the piles they create), and our cognition treats them much as we treat the physical equivalent.

  Labels are, themselves, new objects added to the environment, for which we can then create new labels and pile at even higher levels of abstraction.

  All these points have major implications for what language is to us, especially the last one: our cognition moves from physical-object to abstract-label-object with ease. We create systems with language that we use as additional “built environments.” Labels and categories are certainly enabled by our brains, but they’re not confined there. They are the trellises that shape the way our understanding grows. They are part of the environment that surrounds us, where we recognize them and orient ourselves around them the way we recognize and orient ourselves around landmarks.

  The structure of our language can affect our ability to think through problems and articulate complex ideas. In recent work in composition education, teachers discovered that students were struggling to work through complex subject matter. It was partly because they lacked the linguistic scaffolding for doing so, especially the structural parts of speech that enable complex sentences: words such as “although,” “unless,” and “if.” After those traditional elements were reintroduced, students showed improvement in synthesizing a thoughtful, complex response to an instructor’s questions.[203]

  This further reinforces the idea that language really is infrastructure for us. It gives us the “joints” for bending our thoughts in new directions and connecting them together into new concepts. We use language as part of our cognitive loop to reflect upon and come to new understandings about our environment and ourselves.

  Semantic Architecture

  Language provides not only a scaffolding for the physical world, but another world of language’s own making, created from abstractions that aren’t directly indicative of anything literally in the world at all. We live within the structures and constraints provided by those abstractions just as fully as we do among physical structures.

  Gibson saw language as part of a continuum from simple affordances to highly complex “reciprocal affordances,” saying that when the affordances of vocalization become the semantics of speech—and “manufactured displays become images, pictures, and writing, the affordances of human behavior are staggering.”[204] Affordance of physical information undergirds and enables the workings of semantic function, which then introduces massively higher opportunities for complexity.

  Clark echoes Gibson in saying, “The cumulative complexity here is genuinely quite staggering. We do not just self-engineer better worlds to think in. We self-engineer ourselves to think and perform better in the worlds we find ourselves in. We self-engineer worlds in which to build better worlds to think in.”[205] We don’t have two separate brains—one for language and one for physical things. Even though there is a big difference in the intrinsic nature of physical information versus the mediated nature of semantic information, our cognition does its best to take it all in as one environment.[206] When we say we’re trying to “clarify” a point or “make the complex clear,” it’s not just a metaphor; we’re trying to make an environment’s semantic function make coherent sense to our bodies.

  It can surprise us, the degree to which language establishes structures that change the meaning of our behavior. Take for example an activity many of us encountered in childhood: the staring game.[207] Two people agree on simple rules: “We have to stare at each other, and maintain eye contact, without breaking it; whoever breaks it first loses.” Then, they begin staring at each other, probably trying various tricks to get the other to break contact, like telling a joke or making silly faces. Eventually someone breaks the stare, and the game is over.

  If I walked up to you and just started staring, without this prior agreement, it would be...awkward. But, if we just have a quick verbal exchange about the rules, we create an information structure that we temporarily inhabit together. We’ve built a sort of place that gives us a new context for behavior. Even though it’s only a simple game with a simple rule, it requires mountains of contextual meaning to exist at all.

  Language makes places on its own, but also participates in making and remaking all the other human places we inhabit. The built environment and language are all part of an interconnected system of human meaning-making through using and dwelling.[208] William Mitchell proposes that architecture is as important to language as language is to architecture. “The cognitive function of architecture (distinct from its function of providing shelter) is to create a rich environment for symbol, language, and discourse grounding, and act as the glue of communication that holds communities together.”[209] Language is part of the human-made environment like everything else we build. It establishes structures and rules that we live in together. It creates architecture.

  * * *

  [180] Barrett, Lou
ise. Beyond the Brain: How Body and Environment Shape Animal and Human Minds. Princeton, NJ: Princeton University Press, 2011:214, Kindle edition.

  [181] Pulvermüller, Friedemann. “Brain mechanisms linking language and action.” Nature Reviews Neuroscience July, 2005;6:576–82 (http://bit.ly/10eMjlB).

  [182] Perrone-Bertolotti, Marcela, Jan Kujala, Juan R. Vidal, Carlos M. Hamame, Tomas Ossandon, Olivier Bertrand, Lorella Minotti, Philippe Kahane, Karim Jerbi, and Jean-Philippe Lachaux. “How Silent Is Silent Reading? Intracerebral Evidence for Top-Down Activation of Temporal Voice Areas During Reading.” The Journal of Neuroscience December 5, 2012;32(49):17554–62 (http://bit.ly/1wujhIr).

  [183] Braukus, Michael, and John Bluck. NASA Develops System To Computerize Silent, “Subvocal Speech.” NASA News March 17, 2004 (http://1.usa.gov/1FsgmWJ).

  [184] Deacon, Terrence W. The Symbolic Species: The Co-evolution of Language and the Brain. New York: W.W. Norton & Company, Inc., 2011, Kindle edition.

  [185] Meyer, Robinson. “Researchers Discover the Hot New Technology: Throwing Javelins.” The Atlantic Online December 2, 2013 (http://theatln.tc/1ybTOVw).

  [186] Wilson, Andrew D., and Sabrina Golonka. “Embodied cognition is not what you think it is.” Frontiers in psychology February 12, 2013. doi: 10.3389/fpsyg.2013.00058. See more at http://bit.ly/1shEeC8 and http://bit.ly/1shElO6.

  [187] Deacon, Terrence W. The Symbolic Species: The Co-evolution of Language and the Brain. New York: W.W. Norton & Company, Inc., 2011:354, Kindle edition.

  [188] There are exceptions, depending on how we define noun and verb. The language of Tonga, for example, has a different morphology, but still can be mapped within a noun/verb “prototype framework.” Broschart, Jürgen. “Why Tongan does it differently: Categorial distinctions in a language without nouns and verbs.” Linguistic Typology, de Gruyter, January 1, 1997;1(2) (http://bit.ly/1rnQHEk).

  [189] Chomsky’s theories argue, in part, that the “deep structure” of universal grammar has a logical purity that isn’t always translated to the messy “surface structure” of actual language use. This Platonic-forms approach runs counter to an embodied understanding of cognition, which unifies everything into one, nested, naturally messy system.

  [190] Lakoff and Johnson, Metaphors We Live By. Chicago: University of Chicago Press, 1980:3.

  [191] http://commons.wikimedia.org/wiki/File:Tetragrammaton-Trinity-diagram-12thC.jpg

  [192] Used in their consulting practice since the mid-1990s, but first mention in print in Information Architecture for the World Wide Web. Morville, Peter, Louis Rosenfeld. Designing Large-scale Web Sites. Sebastopol, CA: O’Reilly Media, 1998.

  [193] http://commons.wikimedia.org/wiki/File:Playfair_TimeSeries-2.png

  [194] Posted at toparkornottopark.com by Syliantent, Nikki, June 24, 2014 (http://bit.ly/1CMoWvl).

  [195] I have previously used the phrase “semantic affordance” for this idea, both in early drafts of this book and in “What We Make When We Make Information Architecture” (Resmini, Andrea (Ed.) Reframing Information Architecture Springer, 2014) and various presentations. I have since changed course, to avoid muddling the core value of Gibson’s affordance theory.

  [196] Wilson, Andrew D., and Sabrina Golonka. “Embodied cognition is not what you think it is.” Frontiers in psychology 2013;4(58).

  [197] Wikimedia Commons: http://commons.wikimedia.org/wiki/File:Redtrafficlight.svg

  [198] Photo by author.

  [199] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:263.

  [200] Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Philosophy of Mind) London: Oxford University Press, 2010:44.

  [201] Weick, Sutcliffe, Obstfeld. “Organizing and the Process of Sensemaking Organization Science.” INFORMS 2005;16(4):409–421.

  [202] Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Philosophy of Mind) London: Oxford University Press, 2010:45–46.

  [203] http://theatln.tc/1ybUhH6

  [204] Gibson, J. J. The Ecological Approach to Visual Perception. Boston: Houghton Mifflin, 1979:137.

  [205] Clark, Andy. Supersizing the Mind: Embodiment, Action, and Cognitive Extension (Philosophy of Mind) London: Oxford University Press, 2010, Kindle locations 1424–28.

  [206] Golonka, Sabrina. “Language: A task analysis (kind of).” Posted in Notes from Two Scientific Psychologists, May 25, 2012 (http://bit.ly/1x1ua4A).

  [207] I owe Frederick van Amstel for this example, which he used as an instance of an “interaction” during his presentation at the 2012 Interaction conference in Dublin, Ireland.

  [208] Mitchell, William J. Placing Words: Symbols, Space, and the City. Cambridge, MA: MIT Press, 2005:11, Kindle edition.

  [209] Mitchell, William J. Placing Words: Symbols, Space, and the City. Cambridge, MA: MIT Press, 2005:12, Kindle edition.

  Chapter 10. The Written Word

  To imagine a language is to imagine a form of life.

  —LUDWIG WITTGENSTEIN

  The Origins of Writing

  MOST OF WHAT WE DESIGN INVOLVES WRITING IN ONE WAY OR ANOTHER, and writing has properties that are different from aural communication. Writing is much newer, but it’s no less fundamental to our daily reality.

  Writing as we know it emerged as an elaborate game of charades using scribbled and imprinted signs to create a mélange of evocations—some representational pictures, some phonetic, some a combination of both. Eventually writing became much more about encoding the richness of verbal language than mere pictorial representation, because the pictures were quickly co-opted into representations of the sounds of oral language, instead.[210] That is, pictograms were transformed into phonograms. After all, oral language was already a much more sophisticated and capable ability: why keep using clunky pictures strung together when so much nuance was possible by mimicking the sound of just talking?

  Enter the use of phonetic writing. For example, a picture of a bull with horns might be co-opted to stand for a spoken sound that means “king.” When that innovation happens, the flood gates open: writing starts being used mainly as a way to encode verbal language.[211] Chapter 9 relates how research has shown that our nervous systems fire signals for reading aloud even when we’re reading silently. When we read the written word, we are picking up information from the marks on a surface representing the bodily sounds we make when speaking; and because we’ve been taught (usually from an early age) how to decipher these marks, we have learned how to translate the physical affordance of “seeing marks on a surface” into the mediated meaning we get from “reading.” As we become more practiced readers, we hardly notice that this sound-deciphering is going on. It feels as if we are directly perceiving the meaning on the page.

  What Writing Does

  In the terms of environmental elements, writing changes language from an oral variant event to a written invariant object. And that affords abilities that did not exist previously.

  Oral language is trapped within an event-based, physically constrained mode of experience. Prior to writing, the only way to store and retrieve semantic information was to memorize it. In The Art of Memory (Pimlico), Frances A. Yates famously explains that orators developed elaborate mnemonic methods for retaining long speeches.

  They imagined a huge house with many rooms, and then placed images or objects in the rooms that reminded the orator of the language that is to come next in the oration. “We have to think of the ancient orator as moving in imagination through his memory building whilst he is making his speech, drawing from the memorized places the images he has placed on them.”[212] This strategy of using imagined rooms—physical information about places and connections between them—makes sense, given that we know memory is built up from physical experience of structural patterns in the environment. Yet, even for these incredibly adroit memorizers, the information was only “inside their heads” and could be expressed only orally, part of a linear na
rrative, spoken as an event of sound vibrating the air and disappearing.

  Eventually some of these orators wrote down what they had memorized, which moved the cognitive work from the body to the surfaces of the environment. Writing meant that language wasn’t just a stream of sounds that came and went; it allowed communication to be encoded into actual objects.

  Even in the most advanced digital devices and software, spoken words are under event-based, linear constraints. When I use my car’s GPS, the device speaks the directions, but if I’m distracted and don’t hear them clearly, too bad—they’re gone. I have to look at the written information on the screen (as well as the graphical semantic information, such as the map and status icons) to reference and analyze where I’m going. The entire concept of referencing information or “looking it up” wouldn’t exist without writing.