Understanding Context

by Andrew Hinton

  [232] Photo by author.

  [233] Huelat, Barbara J., AAHID, ASID, IIDA. “Wayfinding: Design For Understanding.” A Position Paper for the Center for Health Design’s Environmental Standards Council, 2007 (http://www.healthdesign.org/chd/research/wayfinding-design-understanding).

  [234] Haverty, Marsha. “Exploring the Phase-Space of Information Architecture” Praxicum (praxicum.com) May 8, 2014 (http://bit.ly/1t9pLgP).

  [235] Wikimedia Commons: http://bit.ly/1xazYaH

  [236] Lynch, Kevin. The Image of the City. Cambridge, MA: The MIT Press, 1960:53.

  [237] Norman, Don. The Design of Everyday Things: Revised and Expanded Edition. New York: Basic Books, 2013:1–2, Kindle edition.

  [238] Photo by author.

  [239] Wittgenstein got the idea from an 1899 article by the early experimental psychologist Joseph Jastrow, who borrowed the figure from Harper’s Weekly (which had republished it from a German humor magazine). The example here is from Jastrow (Wittgenstein’s is a simpler line drawing) (http://socrates.berkeley.edu/~kihlstrm/JastrowDuck.htm).

  [240] Ludwig Wittgenstein Philosophical Investigations Copyright © Basil Blackwell Ltd 1958. First published 1953. Second edition 1958. Reprint of English text alone 1963. Third edition of English and German text with index 1967 Reprint of English text with index 1968, 1972, 1974, 1976, 1978, 1981, 1986. Basil Blackwell Ltd. 108 Cowley Road, Oxford, OX4 1JF, UK

  [241] Jastrow, J. “The mind’s eye.” Popular Science Monthly, 1899, 54:299–312.

  Part IV. Digital Information

  The Pervasive Influence of Code

  PHYSICAL AND SEMANTIC MODES SHAPED OUR CONTEXTUAL EXPERIENCE ALL ALONE, up until the last century. All that time, the invariant structural principles of natural and built environments changed very slowly, if at all. It’s a third ingredient—digital information—whose influence has so quickly disrupted how we experience the other two modes. It’s what has made so many user experience–related fields necessary to begin with.

  Figure IV.1. Digital information

  Part IV explains the origins of digital technology, and why it is different from the other modes. It then explores how digital information influences the way we understand the world, the way we make software, and the properties of digital agents and simulated affordances.

  Chapter 12. Digital Cognition and Agency

  The electric things have their life, too.

  —PHILIP K. DICK

  Shannon’s Logic

  For the realm of information technology, the word information has a specific history. Just as ecological psychologist James J. Gibson chose the word for his work in psychology, Claude Shannon (1916–2001) appropriated it for his own, separate purposes. An American mathematician, electronic engineer, and cryptographer—often called the “Father of Information Theory”—Shannon was the prime mover behind a way of understanding and using information that has led to the digital revolution we’re experiencing today.[242] His work during World War II, and later at Bell Labs and MIT, is foundational to anything that relies on packaging up information into bits (the word “bit” being a conflation of “binary digit”) and transmitting it over any distance.

  One important part of Shannon’s work is how he applied mathematical logic to the problem of transmission, using an encoded (or encrypted) form. Previously, engineers had tried improving the signal of electronic transmission by boosting the power. But that approach could help to only a certain point, at which physics got in the way. Pushing electrons through wires or air over a long-enough distance eventually generates noise, corrupting the signal.

  Shannon’s revolutionary discovery: accuracy is improved by encoding the information in a way that works best for machines, not for humans. This turn goes beyond the sort of encoding seen with the telegraph, where codes were simple patterns of signals corresponding to words, common phrases, or (even more abstractly) just letters. Shannon’s idea had origins in his cryptography work during World War II, when he saw that deciphering a message could be handled by analyzing language rather than semantically.

  Author and historian James Gleick explains how Shannon proposed this approach in a secret paper written during the war, and in so doing, borrowed and recoined the word “information”:

  Shannon had to eradicate “meaning.” The germicidal quotation marks were his. “The ‘meaning’ of a message is generally irrelevant,” he proposed cheerfully. He offered this provocation in order to make his purpose utterly clear. Shannon needed, if he were to create a theory, to hijack the word information. “‘Information’ here,” he wrote, “although related to the everyday meaning of the word, should not be confused with it.”[243]

  By framing the signal of a transmission as a series of discrete (abstract) symbols, it became possible to enhance the accuracy of the transmission by adding symbols that help correct errors; this was extra information against which the receiver can check for breaks in transmitted patterns, or clarify the context of such an abstracted, semantics-free signal stream.[244] This approach is similar to how radio operators will use words like Alpha, Bravo, and Charlie for A, B, and C: in a noisy radio signal, it’s hard to tell letters apart, especially given that many of them sound so much alike. Additional information contextualizes the bits of signal, helping ensure accurate reception.

  So, Shannon’s approach took human meaning out of the enterprise altogether. He took a scalpel to the connection between meaning and transmission, saying in his landmark 1948 Bell Labs paper, “These semantic aspects of communication are irrelevant to the engineering problem.”[245] Shannon redefined “information” much more narrowly as a stochastic construct, built from the most basic logical entity, the Boolean binary unit: yes or no, on or off, one or zero.

  In formulating his ecological view of information, Gibson didn’t discount Shannon’s theories so much as set them aside: “Shannon’s concept of information applies to telephone hookups and radio broadcasting in elegant ways but not, I think, to the firsthand perception of being in-the-world, to what the baby gets when first it opens its eyes. The information for perception, unhappily, cannot be defined and measured as Claude Shannon’s information can be.”[246] Shannon’s approach to information can be “defined and measured” in part because it’s the opposite of human language; it doesn’t emerge through messy, cultural usage. It begins with definition and measurement, from abstract, logical principles, with mathematically clear boundaries.

  From the human user’s point of view, the native tongue of digital things is, by necessity, decontextualized. It doesn’t afford anything for our perception, either physically or semantically, without translating it back into a form that we can not only perceive but understand.

  That significant work of translation has a strong digital influence over the way we think about the world, the way we design and build environments with software, and the way the world around us now behaves. The idea that human meaning is “irrelevant to the engineering problem” is, in a sense, now part of digital technology’s DNA and has a pervasive ripple effect in everything we digitize (Figure 12-1).

  This is why I’m using the word “digital” so broadly, beyond the confines of machine languages and binary code. The more of our world we encode for machines, the more of it is opaque to us, out of our reach, detached from the laws that govern the non-digital parts of our world. We understand our experience by coupling with the information the environment provides us; but digital technology, by its nature, decouples its information from our context. It has forever complicated and changed the way we need to think about design’s communication and craft.

  Figure 12-1. The Digital mode has a strong and growing influence over the other modes of information

  Digital Learning and Agency

  After Shannon’s initial discoveries, information theory didn’t stop at mere transmission and storage. There was another, somewhat more esoteric, area of inquiry going on for several generations: the theory of how machines—using symbolic logic—m
ight do the job of computing. A “computer” had always been a human person, performing the professional role of computing mathematical operations; but human effort often results in human error, and humans can also keep up with only so much computational scale and complexity. So, by the mid-twentieth century, there had been a long-standing interest in ways to automate this activity.

  In addition to Shannon, the work of people such as Alan Turing and Norbert Wiener—prefigured by similar efforts a century earlier by Ada Lovelace and Charles Babbage—led to the creation of machines essentially made of logic itself. Turing, in particular, championed the idea that computing is noncorporeal, not dependent on a particular medium or energy system. He invented the idea of an automated computing machine that functions entirely based on symbols—the Turing machine—that (in theory) could function based on rules built of Boolean, binary fundamentals, from the ground up. Anything that could be represented in mathematical symbols and logic could be computed. Not just mathematical problems, but all sorts of human ideas, questions, and communications—as long as they could be represented in the machine.[247] As a result of this line of inquiry, we now have technology that has agency; the ability to make decisions and take actions on its own.

  This sort of agency has powerful, disruptive effects. Kitchin and Dodge put it this way in their book Code/Space: Software and Everyday Life:

  The phenomenal growth in software creation and use stems from its emergent and executable properties; that is, how it codifies the world into rules, routines, algorithms...Although software is not sentient and conscious, it can exhibit some of the characteristics of being alive... This property of being alive is significant because it means code can make things do work in the world in an autonomous fashion.[248]

  But how does the digital system know anything about that world, which isn’t made of abstractions, but actual concrete stuff? As Paul Dourish explains, we have to create representations of the world with which computers can work:

  Computation is fundamentally about representation. The elements from which we construct software systems are representational; they are abstractions over a world of continuous voltages and electronic phenomena that refer to a parallel world of cars, people, conversational topics, books, packages, and so forth. Each element in a software system has this dual nature; on the one hand, it is an abstraction created out of the electronic phenomena from which computers are built, and on the other, it represents some entity, be it physical, social, or conceptual, in the world which the software developer has chosen to model.[249]

  Human memory is embodied and only occasionally somewhat literal (when we explicitly memorize something). But computer memory works by making exact copies of abstract representations. Computers don’t find their way to abstraction from the roots of physical perception-and-action; they begin with abstraction.

  Every representation has to be intentionally created for the system. This can be done by the people who made the system, or it can be done by some algorithmic process in which the system defines representations for itself. As depicted in Figure 12-2, whereas human cognition emerged from bodily perception and eventually developed the ability to think in terms of abstractions and symbols, digital computing works the other way around. People can go through their entire lives not explicitly defining the entities they encounter. But computers can do little to nothing without these descriptions.

  Figure 12-2. Humans and digital agents learn in different directions

  Writing is already a form of code, with roughly standardized syntax, spelling, and letter forms. So, computers find writing to be a much easier starting point than spoken language. That is, it’s easier to teach a computer a semantic definition of the written word “berry” than it is to teach the computer how to recognize the word when spoken aloud. Speech introduces all sorts of environmental variation, such as tone of voice or regional inflection.

  Although teaching a computer to recognize the spoken word “berry” in some contexts is pretty challenging, teaching it to recognize a picture of a berry is even harder. Sure, we can program it to recognize a specific berry picture, but it really struggles to see any picture of any sort of berry and connect it to “berryness,” which humans tacitly pick up thanks to our embodied experiences with them.

  Computers, however, don’t have bodies unless we add them onto the computing “brain.” Teaching a computer to use a robotic body to find and pick berries is even more complex than visual recognition. It requires definitions not just about the visual qualities of berries, but how to gently harvest something so fragile in the first place, not to mention how to negotiate its body through everything else in the environment.[250] This insight about computers is called Moravec’s paradox, named after AI researcher Hans Moravec, who was one of a group of scientists who articulated it in the 1980s. In Moravec’s words, “It is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility.”[251]

  Of course, since the time of Shannon and Turing, and even Moravec, computers have become much more adept at processing fuzzy ecological and semantic information inputs. Face and voice recognition, street navigation, and other complex pattern-matching capabilities are more possible now with powerful, cheap processors and advanced algorithms. Still, these are extremely limited capabilities, with narrow contextual accuracy.

  This isn’t to say computers will never have embodied learning. There has been some cutting-edge research in biocomputing, which grows computers with organic cells, or even with bodies of a sort, in order to learn more environmentally. For most of us, however, our design work will not involve these exotic creatures. We need to make do with the silicon and bits that are available to us. That means we have to understand the layers of semantic substrate required to make our gadgets do the wonderful things we take for granted.

  We can see this in action when Apple’s Siri attempts (with often hilarious missteps) to understand “where’s the closest gas station?” Here, Siri must rely on the structure of the vocalized vibrations and match them with semantic frameworks that are defined and generated as encoded language. Someone, somewhere, had to use writing to even start teaching Siri how to learn what we mean when we talk to it. Context had to be artificially generated, from already-abstracted inputs.

  And when we use Shazam to recognize a song, the song is not meaningful to the device as it is to us. Shazam is matching the structure of the song with the structures indexed in vast databases. The emotional or cultural context of the song isn’t a factor, unless it’s defined by people in some way. For video content, Netflix has become a market leader in defining these subtle, oblique permutations, which they internally call altgenres, such as “Critically Acclaimed Emotional Underdog Movies” or “Spy Action & Adventure from the 1930s.”[252] The Netflix categories exist only because of enormous work behind the scenes, translating between the nested variety of human life and the binary structures of digital information.

  Everyday Digital Agents

  We’re increasingly giving our environment over to digital agents, programming them as best we can and then setting them loose to do their work. An example of simple digital agency is how my car (a Kia Forte coup) won’t let me perform certain actions in its digital interface if the car is in motion. I was trying to Bluetooth-pair my phone from the passenger seat when I saw this screen. My wife, Erin, was driving. But the system didn’t know that, so it followed the rules it was taught, making the function “Not Available” (Figure 12-3).

  This is similar to the mechanical limitation that keeps me from turning off the engine while the car is still in gear. In both cases, it’s a hidden rule in the system of my car, made manifest by limiting my action. But mechanical limitations can be only as complex as the limits of physical objects will allow. Digital information itself has no such mechanical restriction; it can enact as many thousands of complex rule
s as will fit on device’s microchips.

  Figure 12-3. “I’m sorry, Andrew, I’m afraid I can’t do that.” My Kia Forte, channeling HAL 9000[253]

  That is, digital information has almost no inertia, compared to physical information. If we add to that lack of friction a huge number of more complex, algorithm-based agents, massively disproportionate effects can result. In 2010, the financial world got a bit of a scare when the markets took a momentary plunge of over 9 percent—nearly 1,000 points within minutes, equating to many millions of lost dollars—before recovering most of the drop by the end of the hour (see Figure 12-4). What happened? It turns out that it was due to automated “high-frequency trading,” conducted by computer algorithm. These trades happen much faster than humans could ever conduct business. At the time of the blip, this rapid automated trading accounted for somewhere between 50 and 75 percent of daily trading volume.[254]

  Figure 12-4. The sudden, algorithm-generated dip that shocked the market in 2010 (graph from finance.yahoo.com)

  It came to be called the Flash Crash, and it scared everyone enough to spur investigations and Congressional hearings. Eventually so-called “circuit breakers” were added to systems, but some critics are still wary of high-frequency trading software.[255] And maybe for good reason, since as early as 2012, a “single mysterious computer program” made high-frequency orders and then cancelled them—enough to account for 4 percent of US trading activity in that week. According to one news report, “the motive of the algorithm is still unclear.”[256]