Structure of Speech
We use more than single words or short phrases when we communicate; we need to string together sets of symbols to convey even simple concepts. So a big part of how language works is through its grammatical structure. Without environmental context and the interior structure of language grammar, spoken words are just sounds if they are without significance.
The order of words in a statement determine the statements’ meaning as much as the words themselves. For example, let’s take a look at Groucho Marx’s famous joke from the movie Animal Crackers:
One morning I shot an elephant in my pajamas.
How he got into my pajamas I’ll never know!
It’s old-fashioned humor, and funnier with Groucho’s delivery. But hang on; I’m going to spoil the joke even more:
The poorly structured grammar is what makes the joke funny. It sets us up with a verbally sketched situation that we think we understand to mean he shot an elephant while wearing his pajamas—because that’s the most likely meaning. In the cultural context of the joke, it’s an invariant fact that pajamas are worn by people more than by elephants.
Figure 9-1. An elephant in pajamas (illustration: Madeline Hinton)
The joke works by playing with that cultural invariance, and by structuring the first sentence with what’s called a misplaced modifier—“in my pajamas” is placed in closer proximity to the word “elephant” than the word “I.” We satisfice the way we hear and assume the meaning of the first statement, based on cultural invariance, rather than the indexical proximity between “elephant” and “in my pajamas.” The punch line then completes the statement by shifting the context of the first statement—it turns out the elephant is the one in the pajamas, which is an absurdly silly image that “clicks” into recognition as the correct logical interpretation of the first statement. If he’d said “One morning, in my pajamas, I shot an elephant,” it would have been more accurate, but wouldn’t have been comedic.
Doing such a close reading and analysis of Groucho’s quip certainly takes some fun out of it. However, the exercise illustrates the degree to which meaning depends on context, and how context depends on meaning. It also shows how it’s hard to overthink context—getting it right demands some rigorous analysis.
Understandable language follows invariant conventions of structure, not unlike the physical environment. Language’s emergent structures came from nature the way trees and flowers did; it’s just that humans are the soil where language grows.
Keep in mind that grammatical rules were not bestowed upon us by grammar deities. Syntax emerges from bodies and environments.[187] Our language’s structure is resonant with the structures of human action. The grammatical rules we learn in school are just the patterns that have coalesced over millennia, identified and codified, the same way words emerge from popular usage and are only later identified as new conventions and added to official dictionaries. Language evolves, morphing to meet the changing pressures of its environment.
Just as physical structure happens when two kinds of matter intersect, and layout occurs when there’s a meaningful spatial relationship between various surfaces, language’s structure happens in the intersections and arrangements of parts of speech.
So, it makes sense that language is structured not unlike the elements of the environment we learned in Chapter 6. All languages have the equivalent of object and event—that is, some form of noun and verb.[188] Statements are nested within longer statements and narratives. Depending on the language, syntax (word order and proximity) can be more or less critical for understanding. Yet, all languages depend on some sort of structure, whether it’s provided by modulation of voice, bodily gesture, or grammatical pattern.
Language has to follow its own emergent laws within a given linguistic system; the success of a given utterance depends on its structure and context, its place nested in an environment. Grammar is often behind the most egregious contextual errors in modern life. What works well for Groucho Marx doesn’t work so well in a legal contract or a conversation with your boss. It especially doesn’t work well when we try telling computers what to do—they depend on literal structure even more than we context-interpreting humans.
So much of what we design depends on well-structured language that it doesn’t hurt to consider grammar an important element of good design. Still, the ultimate point I’m making is broader and deeper. We’re now immersed in ambient and pervasive technologies that are essentially made of language. Outside of a carnival fun-house, irony and infrastructure shouldn’t mix. A misplaced modifier can be the equivalent of a bridge collapse.
The Role of Metaphor
An important body of work in the last few decades has been about the connections between the body and language, especially how much of language uses bodily and spatial metaphors. In fact, one of the earliest works connecting language and embodiment theory is the 1980 book by George Lakoff and Mark Johnson, Metaphors We Live By (University of Chicago Press). At the time the book was published, the standard theory (originating in part from linguist Noam Chomsky) was that humans had universal, deep structures that gave rise to language; these structures gave language a formal logic, using repeatable patterns, much like a computer.[189] Language was thought of as disembodied function, and metaphor was considered to be decorative, poetic speech that wasn’t part of language’s core function.
Lakoff and Johnson argued the opposite, positing that language is “fundamentally metaphorical in nature.”[190] Language is the emergent set of behaviors, or techniques, we’ve developed to help us work through and communicate abstractions. The more abstract the concept being expressed, the more that expression relies on metaphor.
Lakoff and Johnson point out many metaphorical uses of the body, such as “give me a hand,” “do you grasp what I’m saying?” or “I need your support; can you get behind me on this?” They explore more sophisticated metaphors involving cultural categories, including how we tend to talk about argument in terms of war metaphors (“Defend your claims”; “Her attack targeted my plan’s main weakness.”).
Of course, there are other metaphors we use in design that aren’t so closely tied with the body, but still make use of conventions learned through nondigital, bodily experience. The personal computer “desktop”—with roots going back at least as far as the 1960s—is a foundational metaphor in graphical user interface design. Even though the desktop doesn’t literally behave in every way like a physical desk, it still provides enough concreteness to help users get started.
Sometimes, though, a metaphor doesn’t quite survive the translation. If we use metaphors inappropriately, it can be confusing. In Apple’s iOS, the category structure for organizing photos borrows from predigital camera-and-film photography. At one level, it nests pictures into the larger container of Albums, and then puts photos into categories within the Albums container, as depicted in Figure 9-2.
Figure 9-2. The iPhone “Albums” structure in iOS 6 that somehow contained a “Camera Roll”
The categories, however, don’t align with how we expect the metaphor of “Albums” to work. For example, the “Camera Roll” is nested under “Albums,” but the metaphor has nothing to do with photo albums; it refers to film cameras that used physical rolls of film in canisters. Film rolls have a limited number of frames to use, and then you have to swap out the used film for a fresh roll. But that’s not how the iOS Camera Roll works either—it has no frame limit other than the memory of the device, and you don’t “swap it out” for a new roll at any point. It just continues to store whatever pictures you leave on the phone.
Interface metaphors don’t need to slavishly copy the physical world, but neither should they appropriate meanings only in the name of seeming familiar, without also behaving according to the expectations they set. When the labels present a nested structure that’s actually the opposite of the physical referents they borrow from, one wonders why use the metaphors at all.
Visual I
nformation
Graphical information is also semantic but uses iconographic and indexical approaches in ways for which words are not as well suited. Visual information is especially good at borrowing from the objects of the physical environment to create explanations, metaphors, and spatial arrangements for conveying meaning.
Sometimes, graphics are used in strictly an iconic manner. Photographs, realistic paintings, and even abstract images, such as the stairs sign illustrated in Chapter 8 (Figure 8-2), can be used as icons for what they depict. Graphical user interfaces make heavy use of iconography, often more as symbols than strict icons. Some visual metaphors are for functions that have no present, physical referent. Figure 9-3 presents an example from a Macintosh computer: a padlock’s open or closed state represents whether an administrator has unlocked settings so that she can make changes to settings.
Figure 9-3. The locked and unlocked states in an OS X dialog box
The icon (and the mechanical sounds the computer makes when activating it) brings a representation of physical information into a semantic display. It clarifies, with body-familiar imagery, the state of locked versus unlocked. Of course, there is no actual padlock in the computer. Digital information is abstract by nature. Therefore, it requires translation, such as these metaphors. Even before the invention of graphical interfaces, computers used similar metaphors but with typed commands, such as “get,” “put,” and “set.”
We also use graphics in less literal ways to physically represent abstract ideas. An early example is a diagram called the tetragrammaton (see Figure 9-4), which represented the Christian Holy Trinity.
Figure 9-4. A tetragrammaton from the twelfth century[191]
It takes something that people could not see in the physical world—the relationships between the Father, Son, and Holy Spirit, together in a single deity—and puts those together into a shape illustrating how the Christian God can be three beings and one being at the same time.
Even now, we use similar diagrams to explain abstract concepts, because making the ideas into representations of physical objects makes them easier to grasp. For example, the Venn diagram shown in Figure 9-5 shows the intersection of mathematical sets, or the shared qualities among multiple entities. It does a great job of making the invisible visible, working with otherwise disembodied ideas as if they were concrete objects.
Graphics are especially useful for giving form to the abstractions of mathematical measurement. The first known usage of visual explanation for data was by William Playfair in the late eighteenth century, as shown in Figure 9-6. The figure shows a trade balance relationship over time, involving England, Denmark, and Norway.
Figure 9-5. The “three circles of information architecture” introduced by Rosenfeld and Morville in the 1990s[192]
Figure 9-6. William Playfair’s time series of exports and imports of Denmark and Norway[193]
In addition to this line-graph method, Playfair went on to invent the bar chart, the pie chart, and the circle graph—essentially creating the field of graphical statistics.
A recent practical example is the concept for a parking sign portrayed in Figure 9-7. Instead of the confusing jumble of words and numbers we usually see, the designer represented time spatially.
Visual information lets us model abstraction and work with thoughts and concepts symbolically, while managing to provide objects we can see, manipulate, and arrange. Like writing, this capability makes it possible for us to work with more-complex systems, over greater scale and longer periods of time. Even though our main focus is how words create information, they are often assisted by graphics, and vice versa. All of it is semantic information, and all of it functions as structure we add to our environment.
Figure 9-7. A graphical parking sign, by designer Nikki Sylianteng[194]
Semantic Function
How does semantic information work for perception and cognition? If we reserve Affordance to mean physical information’s direct specification of physical opportunities for action, where does language fit into the model?
Language is fundamental and clearly affects our behavior and experience far beyond mere abstraction. Signifiers are all fine and good, but signification is too easily construed as a cloudy, disembodied concept. Although accurate, it runs the risk of detaching how we think of language from its truly visceral effects. At the same time, language is not the same as physical information. The word “hammer” doesn’t budge a nail, not even a smidgen.
I’ll be using the phrase semantic function to indicate the way we use language as part of our environment (see Figure 9-8). What I hope it conveys is that semantic information has a real, functional role as invariant structure in our surroundings. Even though language can be exceedingly abstract—as in a word like “love”—it can have real, physical consequences—as when our hearts can suddenly race when we hear, “I love you.” Language can be a sort of civil machinery, such as laws and contracts. It functions as instruction, like virtual guardrails for baking a cake or driving a car. So, semantic function is the near-equivalent of physical affordance but for semantic information.
From the perspective of the human agent, language and semantic conventions can become tacitly understood parts of the environment, to the point where the agent uses them with nearly the same familiarity and facility as the physical parts of the world.[195] In other words, to the agent, there is only information from the environment. The soft assembly of the agent’s experience uses semantic function and physical affordance indiscriminately, in whatever combinations necessary.
As designers, we have to assume that for users there is no meaningful affordance-versus-function separation.[196] From the user’s point of view, the main differences are along the explicit-to-tacit spectrum. But, to make something that users can easily use, we have to do the hard work of understanding these distinctions between affordance and semantic function in the work of design. Designers will likely continue using the term Affordance more broadly than I am specifying here. But if these physical-versus-simulated dimensions are not clarified in the work of design, it can lead to dangerous assumptions about what users perceive and understand.
Figure 9-8. Semantic function surrounds human perception and augments physical affordance
The red stoplight in Figure 9-9 is a physical object that emits light in three colors. In the context and learned conventions of roadway driving, its red mode means Stop. Most of us respond to it physically, stopping in front of it. When learning to drive, we have to think about it more explicitly, but eventually we respond to it with little or no conscious attention. It becomes tacitly picked up, indirectly controlling part of our environment. Is it as directly perceived and controlling as a physical barrier? No, but it’s about as close as a semantic sign can come to being such a barrier.
Figure 9-9. A traffic light, displaying red to mean “stop”[197]
For a twenty-first-century person in the developed world, a huge portion of the environment is made up of these signs and symbols. They surround us as bountifully as trees, rocks, and streams once surrounded our ancestors.
Software interfaces are made entirely of semantic conventions. The only truly physical affordance on a digital device’s screen is the flat surface of the screen itself. The only way it can interact with us is through signs and symbols—words with simulated surfaces and objects on the screen, or sounds whose meanings also require learning as language. There is ongoing research to create haptic interfaces that mimic the contours of three-dimensional surfaces, but they will only enhance the simulation. We will still need to learn what the simulated button, edge, or other object actually does.
That’s because any object that controls something beyond its present physical form works a lot like language. Consider a typical light switch, such as the one in Figure 9-10. Even this simple mechanism might not be immediately clear in its function to someone who has never encountered one before. But for those of us who grew up around the use of such switches, there is intrin
sic physical information that specifies the affordance of “flipping” up or down. That is all we know from looking at the object alone. What does the switch turn off or on? Is that even what it does? The answers depend on contextual relationships.
Figure 9-10. A domestic light switch[198]
To know what this switch will do beyond its intrinsic structure, I have to either be told, or I have to flip the switch to learn what happens. I’ve often been startled by the angry growl of a garbage disposer in a sink when I expected to illuminate the kitchen. To really understand controls like this, we often resort to adding labels, or otherwise creating semantic context between the object and its ultimate effect.
The switch is acting as a symbol. It’s a signifier that could mean almost anything. Like language, our technological systems depend on contextual learning and associations. Yet, as we do with all familiar language, we conflate these elements. If you asked, “What does that switch do?” and I answered, “It flips up and down,” you’d think I was joking around. “Of course it flips up and down, but what does it do when you flip it?” you’d counter.
When we point at a switch to ask someone “can you turn on the light?” when the light is actually above our heads, we’ve merged the signifier and signified across space. We’re using one context to talk about and control another, essentially making them one. If we were designing a new light-control system, we would want to untangle the semantic function and physical affordance dynamics at work here, because they would inform how we might improve the system for use. At some point, though, we would have to again see it as one conflated, intermingled, nested system, because that’s how its users will need to perceive it, as well. This conflation happens when we learn anything, from how to use a computer mouse or game console controller, to how we learn to swipe left to right to unlock our smartphones.
Understanding Context Page 17