Understanding Context

by Andrew Hinton

  For example, across platforms, actions can have significantly different consequences: in one mobile app, double-tapping a photo zooms in on it, whereas in another, the same gesture marks the photo as “liked,” even if you find the photo horrible or offensive. Even within a single platform, the rules can radically change. In Facebook, it seems every few months the architecture of privacy morphs again, disrupting the invariants users learned about who could see what information and when. This is not unlike waking up one day to find the roof removed from one’s house. If these invariants were created by physical affordances, they couldn’t change so radically. We know software isn’t physical. But because perception depends upon invariant structure, we see it and use it whenever it seems to be offered to us, even if a software code release can upset those invariants in a moment’s time.

  These differences and disruptions are not the sorts of behaviors we evolved to perceive with any accuracy. They are instead the black ice, or quicksand, of digital objects and places.

  The Principle of Nesting

  All these invariant components of the environment are perceived in relation to one another, and the principle by which we perceive these relationships is the ecological principle of nesting. Animals experience their environment as “nested,” with subordinate and superordinate structures.

  Nested invariants establish the persistent context within which motion and change happen, both fast and slow. As Gibson says somewhat poetically, “For terrestrial animals like us, the earth and the sky are a basic structure on which all lesser structures depend. We cannot change it. We all fit into the substructures of the environment in our various ways, for we were all, in fact, formed by them. We were created by the world we live in.”[123] The way the world is nested forms a framework for how we understand it.

  Within the earth and sky, at subordinate levels, there is more nesting: “canyons are nested within mountains; trees are nested within canyons; leaves are nested within trees. There are forms within forms both up and down the scale of size.”[124] Nesting provides a framework for the agent to pick up information about the environment at different levels of scale, focusing from wide to narrow, huge to tiny.

  Even without telescopes and microscopes, humans have a great deal of environmental information to pick up; the ability to learn an environment (and therefore, learning in general) might be largely due to the ability to break it down into loosely defined tiers and components, none of which exists discretely in a vacuum.[125]

  Nesting versus Hierarchy

  Nesting is not the same as hierarchy (see Figure 6-1), because it is “not categorical but full of transitions and overlaps. Hence, for the terrestrial environment, there is no special proper unit in terms of which it can be analyzed once and for all.”[126] Again, this is a property of how terrestrial creatures such as humans perceive the environment, not an artificially defined property like “meter” or “kilogram.” It isn’t about logically absolute schemas such as the periodic table of elements, or the mutually exclusive branching we see in strict hierarchies.

  Figure 6-1. Hierarchies (left) are categorically defined, with in an objective parent-child tree structure, separate from the perceiver. Ecological “nesting” (right) has elements contained within others, but in perceiver-dependent fashion that can shift and overlap depending on the present perceptual needs of the individual.

  A conventional hierarchy is about categorical definition that’s true regardless of context, whereas our ecological experience of the environment shifts depending on circumstances. A geological definition of “cave” doesn’t change with the weather. For an animal, however, a cave that provides excellent protection in dry weather might be useless during the rainy season, if its opening is angled to let water in: suddenly “inside” feels like “outside.”

  Likewise, the way we think of “here” and “there” can shift—which on the face of them seem like clear opposites. But, these designations are dependent on the nested context. You might be sitting at home, in a comfortable reading chair, from which you can see a bookcase on the far wall. You’re trying to remember if you left a particular book at work and wonder if the book is there or if it is here at home—which could mean anywhere in the house. You look up to the bookcase and see it, and realize with relief that, yes, not only is it here at home but here in the room with you, not there in some other room. Still, when you decide you want to read the book, it’s no longer here but there on the bookshelf, because now here and there are defined not by mere location but by the factor of human reach. From a geometrical point of view, nothing changed, but from an ecological point of view, many shifts in perceived structure happened in a matter of a few moments.

  Digital Environments and Nesting

  Interfaces can graphically represent these nested relationships with boxes and panels, outlines and background shading, with one set of information contained by another, and so on. Information architectures (which depend in part on interfaces representing this containment) also present nested structures through categories and semantic relationships.

  Yet, interfaces need ways to provide invariants while still allowing users to move through structures from various angles based on need; a user might shift between different nested perspectives in the course of only a few minutes.

  For example, Figure 6-2 presents a scenario in which an office worker might need to gather information about a prior project, including its documentation, personnel, and manager involvement. The worker might begin with his focus mainly on finding a related document, from which he discovers the name of the document’s author. He then uses the author to find who the team manager was, in order to find others who might have past knowledge of the project.

  Figure 6-2. Information is nested from different perspectives for the same user, at different points in an activity

  In this case, the user’s information pickup organizes itself around documents and projects first, then finding people by name, and then finding people by organizational relationship. All these structures overlap in various ways: a document can be just a document or part of a project, and the project might be related to a larger ongoing enterprise program but also related to various teams in several departments, which themselves can shift in importance depending on a worker’s cognitive activity from one task to the next.

  In particular, for the work of information architecture, we have to remember that users might comprehend an information environment in a nested way, but that doesn’t mean they naturally comprehend artificial, categorical hierarchies. Learning a new hierarchy of categories depends on explicit, conscious effort. But an information environment that works well at an ecological, embodied level might seem incoherent through the lens of logical hierarchy. When online retailers add holiday-specific categories to their global navigation menus, it upsets the logic of hierarchy, but it makes perfect sense to the many shoppers who are seasonally interested in holiday gifts.

  Facebook users experienced disorientation, especially in its early years, as it expanded from being nested within a single university to multiple universities, and then to high schools, and finally opened up to everyone. Likewise, its internal structures (and its invisible reach into other contexts, such as with Beacon) are continually “innovated” to the point at which users struggle to grasp where they and their actions are nested in the shifting fun-house of illusory invariants.

  Surface, Substance, Medium

  When we perceive information in the environment, we’re largely perceiving information about its surfaces, which are formed by “the interface between any two of these three states of matter—solid, liquid, and gas.” Where air meets water is the surface of a pond. Where a river meets earth is a surface is a riverbed. Where air meets earth is a surface—the ground—which, for terrestrial animals, is “both literally and figuratively” the ground of perception and behavior.[127] The structure of the environment is, in essence, its “joinery” between surfaces and substances. These are the “seams” of o
ur world. If it were “seamless,” there would be no structure, and no perception.

  These surfaces are made of substances which are substantial enough that, when environmental energy (sound, light, and so on) interacts with them, the resulting information can be perceived. We didn’t evolve to readily perceive invisible things or things that are too tiny or huge to apprehend, at least as far as physical information is concerned.

  Medium refers to the substance through which a given animal moves. For a fish, it’s water. For terrestrial creatures, it’s air.[128] Another quality of a medium is that it physical information can be detected through it. Solids do not allow the full pickup of information because they impede ambient energy.[129] Sure, we can sometimes hear noise through a solid wall, but only because there is air on both sides of the wall that reverberates the sound. Fill our ears and the room around us with wall plaster, and we’ll hear a great deal less.

  Digital Examples

  In software, we see users trying to figure out what parts of a system they can move through (medium) versus parts that do not afford action (impeding substances/surfaces). An understandable interface or information architecture provides invariant indications of where a user can go and how. Perception evolved among actual substances and surfaces, made of atoms, so even in the insubstantial realm of language and bits, it still reaches for substantial information, hoping something will catch hold.

  Objects

  An object is a “substance partially or wholly surrounded by the medium.” Objects have surfaces but with a topology that makes them perceivable as distinct from other surfaces.

  Some objects are attached and others detached. An attached object is a “protuberance” that has enough of its own independent surface that it can be perceived to “constitute a unit.”[130] It’s topologically closed up to the point at which it is attached to a surface. An attached object can be counted, named, thought of as a thing, but still understood to be a persistent part of the place to which it’s attached. An attached object can’t be displaced unless it’s detached by some kind of force (which, once applied, then results in a detached object). Attached objects are in an invariant location, so they make up part of the invariance of the surrounding information.

  Detached objects are topologically closed, independent entities. Learning to perceive a detached object is different from perceiving an attached one because it has different properties and affords different actions. A tree branch might be attached to a tree, but when we break it off, it is now detached, affording actions unavailable in its attached state. The many bumpers of a pinball machine and other structures are attached objects; the pinball is detached.

  Recall how we established earlier that when we pick up an object and use it as a tool with which we extend our body’s abilities, we perceive it as an extension of ourselves.[131] A tree branch can go from being an attached object, to a detached object, to part of “me” with great ease.

  Phenomenology and Objects

  In some ways, embodied-cognition science has been catching up to ideas that have been explored in other disciplines for at least a century. For example, phenomenologist Martin Heidegger famously described three modes of experiencing the world:

  Readiness-to-hand

  When an experienced carpenter is using a hammer, the carpenter doesn’t think consciously about the hammer. Instead, the hammer becomes a natural extension of the carpenter’s action, and the carpenter “sees through” the tool, focusing instead on the material being affected by hammering. In our levels of consciousness spectrum, this is tacit, unconscious action, in which there’s no explicit consideration of the tool any more than explicitly considering our finger when we scratch an itch.

  Unreadiness-to-hand

  When the action of hammering is disrupted by something—a slippery handle, a bent nail, a hard knot in the wood—the hammer can become “unready-to-hand.” Suddenly the carpenter must stop and think explicitly about the tool and the materials. The hammer can become less of an extension of the self at this point—no longer “seen through” but considered as an object with a frustrated or interrupted potential as a natural extension of the body.

  Presence-to-hand

  When looking at a hammer as “present-to-hand,” we remove it from its context as a tool or a potential extension of the self; instead, we consider it neutrally as an object with properties of size, density, color, and shape.[132] This is explicit consideration outside of the cycling between the “unreadiness” and “readiness” in the context of the tool’s affording functions. As designers, we often need to switch into this level of awareness of what we are designing: its form and specification and other aspects that the user doesn’t (or normally shouldn’t) have to worry about.

  Manipulating and Sorting

  A detached object can also be manipulated, considered, and organized in ways that attached objects can’t. It can be compared with another object, grouped with similar ones, and sorted into classification schemes, allowing counting and assignment of a number.[133] As we’ll see, this is what we do with the abstractions of language, with which we use words or pictures to categorize and arrange the world with signifiers rather than the physical elements themselves.

  Objects with Agency

  Among the world’s detached objects are animals, including humans.[134] We evolved to comprehend that these objects are special, and have a sort of agency of their own, an animus evident in their physical actions, signals, and (for humans) symbolic language. In modern life, many of us don’t encounter a lot of wild animals, but we’re familiar with pets as independent agents in our midst. Other humans interface better with us than animals (usually...), because we share more of an umwelt (environmental frame) with them, including the full capabilities of human language. Regardless of human or animal, we spend a lot of energy figuring out what these agents are up to, what they must be thinking, or what they need.

  Of course this means we ourselves are detached objects, and in moments of objective consideration, we sometimes perceive ourselves as such. Typically, though, we don’t see ourselves with the objectivity that other agents perceive us. So, a human agent might be puzzled or troubled when an environment treats him like another object rather than as a “self.” When we find ourselves in bureaucratic labyrinths, shuttled along as if on a conveyor belt, it feels “dehumanizing”; we feel the same when digital systems communicate with abrupt sterility, or coldly drop us from what felt like a meaningful exchange. Technological systems tend to lack contextual empathy.

  Digital Objects

  Digital objects might be simulated things on screens or physical objects with digitally enabled properties.

  Simulated objects are, for example, icons in a computer’s interface. From early in graphical-user-interface (GUI) design, the convention has been that these should be able to be moved about on a surface (such as the desktop) or placed in other container objects (folders). However, these are flat simulations on a flat screen. If they were physical objects, we would be able to determine through action and various angles of inspection whether they were attached or detached, or if they were merely painted on a surface. Digital displays don’t give us that luxury, so we have to poke and prod them to see what they do, often being surprised when what seemed an interactive object is really a decorative ornament. Like Wile E. Coyote in the Road Runner cartoons, we slam into a wall where the tunnel is only painted on.

  Software also uses text as objects of a sort. Labeling our icons means we’re attaching word objects to the picture objects. But even in something like a text-based email application, or a command-line file-transfer-protocol (FTP) interface, for which there’s little or no graphical simulation, there are still commands such as “put” and “get” and “save” that indicate we are treating chunks of text as singular objects to be moved about, modified, put away, or retrieved from storage.

  Sometimes, we perceive an entire interface as an object, with buttons and things that we are manipulating the way w
e would manipulate a physical object such as a microwave or typewriter. These objects are made up of other, smaller objects, nested together into compound objects (with compound invariants). Other times, we perceive an interface as a place (which we will discuss shortly). And often, because of the weird nature of digital information, we perceive a digital thing as both object and place at the same time, which can be just fine or very confusing, depending on how appropriately it’s expressing itself as one and the other, or the perspective the user brings to it.

  As David Weinberger explains in Small Pieces Loosely Joined: A Unified Theory Of The Web (Basic Books), this has been the case with the Web for a long time: “With normal documents, we read them, file them, throw them out, or send them to someone else. We do not go to them. We don’t visit them. Web documents are different. They’re places on the Web....They’re there. With this phrase, space—or something like it—has entered the picture.”[135] Our everyday language shows how this works: we find ourselves using prepositions interchangeably in digital environments, such as “you can learn more on Wikipedia” or “I’m in Facebook right now” or “find my book at Amazon.com!” Physical structures tend to constrain the prepositional phrasing that makes sense for them. I don’t say I am “on” my house if I am inside it, but I might say I am “in” or “at” my house. Digitally driven structures are more fluid, allowing us to use words with more flexibility, based on the context of the interaction.