Understanding Context

by Andrew Hinton

  This is the world we live in now, where we have to guess at the motivations not of Olympian gods, but of computer-process daemons whirring away in the nervous system of our global economy. Digital agents are natively mathematical creatures, unconcerned with slowing down enough for us to keep up, or being transparent about their inner dynamics, unless we design them to be otherwise.

  Although it takes specialized knowledge to create a market-trading algorithm, laypeople increasingly have access to tools for making their own decision-making agents. For example, IFTTT—If This Then That—is a cloud-based service with which users can create “recipes” for simple procedural connections between other online services (see Figure 12-5). For example, “If I post a picture to Instagram, store a copy in my Dropbox archive.” IFTTT is like a virtual Lego set made of semantic functions with which we can create our own digital helpers. It provides active digital objects we can use to conditionally invoke events, interwoven with defined places—and increasingly with physical objects.

  IFTTT has started adding new triggers for physical devices, such as wearables and smart home products.[257]

  Figure 12-5. Recipes from IFTTT.com

  These new tools change the nature of the places they’re connected to, whether an online place such as Instagram, or a physical place like the home, or an object such as one’s own body.

  IFTTT has made a nicely understandable system that enables a lot of complexity with simple rules. Still, these invisible agents can scale only so far until an average person will have trouble keeping up with them all. Memory typically requires repeated exposure to a perceivable pattern, but the purpose of most such agents is to “set it and forget it.” There are already many common set-and-forget services, like online subscriptions to medicine and groceries, or automated bill-pay withdrawals—and these will seem primitive and few in another 5 to 10 years. It leads us to question what will help us manage all this personal automation. Will we need agents for keeping track of our agents?

  Ontologies

  Computers are machines that do not need human-understandable context to function within and among themselves. Sure, we created them, but there is no intrinsic requirement that once made, they ever have to provide an output that we can comprehend. Early computers required their users to interpret hole-punched cards and patterns of blinking lights to read their computed results. For example, the 1970s Altair hobbyist computer (Figure 12-6) had only switches and red LEDs on its front panel as input and output mechanisms for entering and reading the results of simple programs.

  As computers became more powerful, they could be used to store and process more-complex concepts. But those concepts have to be described and modeled for the machine to work with them. To accomplish this task, information science makes use of an old concept in a new way: something called ontology.

  Figure 12-6. The Altair 8800b user interface[258]

  For centuries, ontology has referred to the philosophical study of being, existence, or reality. It also has to do with the categories of things that exist, and the relationships between those categories. This conceptual sort of ontology is part of an ancient branch of philosophy called metaphysics. Language’s role in ontological questions has increased in significance over the centuries, as philosophers (and anyone else concerned with language) have come to realize that language and “being” are fundamentally connected. What does it mean to say, “There is a calendar hanging on the wall, and I know it is there, and I know it has information on it that means X or Y”; and what does it mean to say, “We both exist in the same place and are experiencing the same calendar?” To consider the question fully requires talking or writing about it; ontological questions are, after all, questions.

  Ontology is also a term used in information science for a related, but different meaning: the formal representation of a set of concepts within a domain, and the specification of the relationships between those concepts. That’s a mouthful, but all it really means is the way we teach digital machines how to understand the inputs we expect them to process. A business software platform such as Microsoft Outlook has data objects that must be defined so that the system knows what its calendar is and what functions are expected of it. Google’s search algorithms and massive semantic databases have to know what people mean when they search for things related to calendars—all the synonyms and related products, conversations, or anything else that has to do with how we use the word “calendar” in English-speaking culture.

  Sophisticated information systems require these sorts of definitions in order to make sense of the work we give them to do. Ontologies stand as a digital system’s version of invariant structure for its nested semantic environment—the lens a computer uses to process human ideas and entities. For example, the BBC has a publicly available set of ontologies (see Figure 12-7) it uses for structuring its information, including what it means when it refers to a television “series” versus an “episode,” or what it means to publish a “story” and its various related components.

  Figure 12-7. The “programme” ontology from the BBC[259]

  In short, this is a foundation for how we teach digital systems about human context. There’s plenty of digital information that computers share that is only about computing, so it doesn’t require semantic context. Yet, we made computers to be part of the human environment, so most of what they do eventually needs to be translated into signifiers that people can comprehend.

  When designing systems for people to use, we should consider both the conceptual and formal definitions of ontology, because much of our work is about creating a bridge between those poles. Looking at the Google Calendar example in Chapter 11, we can see how ontology is at the center of the problem. What is “Calendar” in the Google ecosystem? In some places, the word is used to indicate the simulated physical calendar we see on a screen. In other instances, however, it indicates the subscribed calendar-feeds that are only subsets of the entire calendar environment. The digital layer has clear definitions for what these calendars are, because each construct has a machine-friendly name and description. But to the end user, they’re all just “calendar.”

  Google had a related ontological problem in 2010 with the service it called Buzz, which attempted to integrate social networking into the successful Gmail platform. When Buzz rolled out, it created a prepopulated list of “friends” for every user, based on measurements gathered from users’ Gmail history, such as frequency of contact.

  Upon launch, many Gmail users were surprised to discover this new set of rules and structures that changed the meaning of their Gmail environment, which went from being about email to being about email-plus-something-else that wasn’t entirely clear to them. The biggest problem was this: Buzz automatically decided for users just who those friends were versus people they didn’t want in such a list, such as bosses, coworkers, or hostile ex-spouses. The system also, by default, publicly showed your list of friends—letting everyone know the people with whom you communicate most.[260] These missteps cost Google, among other penalties, 20 years of auditing by the United States Federal Trade Commission.[261]

  Buzz was working from an ontology that defined “friend” too simplistically, walking into the same trap that Facebook’s Beacon had tripped only a few years earlier. “Friend” is nested in our lives, not a rigidly hierarchical entity. We might refer to someone as a friend in one circle of people but in another circle say that person is an acquaintance or partner. However, Buzz created a digital agent (based on an algorithm) that saw the world through a narrow spectrum that was efficient for the machine’s perspective but disastrously wrong from the human point of view.

  This happened even though Buzz was tested extensively within the huge corporate social context of Google before it was launched.[262] That prelaunch context was misleading, though, because a corporate environment has tacit cultural norms that make it a very different place compared to nonwork life. Add to this problem the fact that users had already become used to Gmai
l as an environment with predictable, invariant qualities. Buzz introduced new rules and structures, changing the architecture of how Gmail worked. It didn’t just add features—it changed the nature of the environment fundamentally.

  There are information-science specialists who can create ontologies that work well for machines, and there are philosophers who continue to explore what it means to “be.” But, for those of us making user-facing environments, ontology is about establishing understandable semantic function that solves for the contextual gap between person and machine.

  * * *

  [242] The amazing, fascinating history of information theory and figures like Shannon is beyond our scope, alas. For a delightful telling of these stories, read James Gleick’s The Information: A History, a Theory, a Flood.

  [243] Gleick, James. The Information: A History, a Theory, a Flood. New York: Random House, Inc., 2011, Kindle locations: pp. 3848–53.

  [244] Gleick, James. The Information: A History, a Theory, a Flood. New York: Random House, Inc., 2011, Kindle location: 3922.

  [245] Shannon, C. E. “A Mathematical Theory of Communication.” Reprinted with corrections from The Bell System Technical Journal July, October, 1948;Volume 27:379–423, 623–56.

  [246] Gibson, 1979, p. 24

  [247] For an excellent overview of how Turing’s ideas intersect with Gibsonian ecological theory, see the section called “When is a Turing Machine Not a Turing Machine” in Louise Barrett’s Beyond the Brain (2011). Also, see the following writings by Andrew Wells: Wells, A. “Gibson’s affordances and Turing’s theory of computation.” Ecological Psychology 2002; Volume 14: 140–80, and ———. Rethinking Cognitive Computation: Turing and the Science of the Mind. London: Palgrave, 2006.

  [248] Kitchen and Dodge, 2011:5.

  [249] Dourish, Paul. Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA: MIT Press, 2004:137, Kindle edition.

  [250] In fact, new embodied paradigms in robotics have resulted in robots that gracefully navigate all sorts of terrain, such as the naturally limbed but small-brained “Big Dog” robot designed by Boston Dynamics, a company recently acquired by Google (http://www.bostondynamics.com/robot_bigdog.html).

  [251] Moravec, Hans. Mind Children. Cambridge: Harvard University Press, 1988:15.

  [252] Madrigal, Alexis C. “How Netflix Reverse Engineered Hollywood” The Atlantic (theatlantic.com) January, 2015 (http://theatln.tc/1sKXxHF).

  [253] Photo by author.

  [254] “Surge of Computer Selling After Apparent Glitch Sends Stocks Plunging.” New York Times May 6, 2010 (http://nyti.ms/1uzHTNF).

  [255] http://en.wikipedia.org/wiki/2010_Flash_Crash

  [256] “Mysterious Algorithm Was 4% of Trading Activity Last Week.” CNBC.com, Monday Oct 8, 2012 (http://www.cnbc.com/id/49333454).

  [257] https://ifttt.com/jawbone_up and https://ifttt.com/wemo_motion

  [258] Wikimedia Commons: http://bit.ly/1xaAIws

  [259] http://www.bbc.co.uk/ontologies/

  [260] Carlson, Nicholas. “WARNING: Google Buzz Has A Huge Privacy Flaw” Business Insider (businessinsider.com) February 20, 2010 (http://read.bi/1rpOyrE).

  [261] Wouters, Jorgen. “Google Settles With FTC Over Privacy Violations on Buzz” DailyFinance.com, March 31, 2011 (http://aol.it/1vHPCgF).

  [262] Krazit, Tom. “What Google needs to learn from Buzz backlash” CNet.com, February 16, 2010 (http://news.cnet.com/8301-30684_3-10454683-265.html).

  Chapter 13. Digital Interaction

  I don’t design stuff for myself. I’m a toolmaker. I design things that other people want to use.

  —ROBERT MOOG

  Interfaces and Humans

  WHEN WE INTERFACE WITH DIGITAL SYSTEMS, we’re doing so through many layers of abstraction, so it’s necessary to provide environmental elements that we can recognize and understand. That’s essentially what computer interfaces are: artificial environments that bridge the gap between digital information’s total symbolic abstraction and our perceptual systems’ need for affordance, whether physical or simulated.

  It’s easy to forget that the word “interface” isn’t necessarily about people. For many years, the word mainly had to do with how one machine interoperates with another. For example, an API is an application programming interface with which software engineers can make two applications share functions and data; and the acronym SCSI means Small Computer System Interface—a hardware standard for connecting devices and peripherals such as hard drives and personal computers (see Figure 13-1, left). Like most things related to digital systems, software and hardware interfaces work best when they are rigorously defined and kept to an efficient minimum, such as with a keyboard (Figure 13-1, right) or mouse. Overlapping, extraneous, or ambiguously defined interfaces are anathema to efficient, reliable digital system design.

  Figure 13-1. Left: A “terminator” for a Small Computer System Interface, or SCSI (pronounced “scuzzy”), device chain;[263] right: a keyboard for the structured-language input from human fingers[264]

  Both of these are hardware interfaces for sharing data between two systems. But in the keyboard’s case, one of the systems is human. As we’ve seen, humans perceive and act in ways that are abundantly ambiguous and overlapping; we satisfice our way through our activities, and only tacitly and passively comprehend most of what we encounter. That is, humans are horribly inefficient, irrational systems. In spite of this fact, it’s perhaps ironic that software-making organizations expend so much time on digital-to-digital interfaces, and so little on the interface between the digital system and the human, who is by far the most challenging, complicated system in the mix.

  At their best, digital systems accommodate our physical and semantic informational needs quite nicely; at their worst, they require us to think and behave like computers, which is hard work for us squishy-brained animals. It’s in this translation between digital and physical/semantic modes where we find the seemingly infinite varieties of architectural and design challenges that sparked the need for new fields of practice, like human-computer interaction, usability engineering, interaction design, information architecture, and others.

  In only about 50 years, digital interfaces have gone through a rapid evolution, from an age of punch-card-driven mainframes to our current plethora of input methods that use high-resolution graphics, virtual keyboards, voice recognition, and motion-tracking sensors. There are other resources beyond this book for exploring all the varieties of human-computer interfaces and models of interaction with digital systems. Although we will touch on some specific interaction examples, our purpose here is a bit broader: to establish that interfaces are part of the environment we inhabit and are themselves smaller environments nested within the larger ecological context.

  For a generation, software was made mainly for trained specialists in rarified environments, performing highly structured tasks. Now that most software is being made for regular people doing everyday stuff, software has to be reframed as everyday environment that laypeople expect to use as they do everything else around them—mailboxes, toasters, elevators, and bridges. This is the place of most software now—devices in the world that need to just work, without a lot of fuss.

  Moreover, when we create digital agents, we have to do the extra work of translating between their black-box nature and the cognition of regular people. Most of this translation is done by using language.

  Sometimes we get pretty close to success, but not close enough—as is demonstrated in the example of a common gas pump. Gas pumps use digital information to handle transactions between the pump, the store, and credit-card processing services. They’re simple digital agents, but they are agents nonetheless—making decisions based on established rules in their software. When I tried pumping gas recently, I’m embarrassed to admit it took me a full minute to realize there was a sticker above the digital display (Figure 13-2) translating the passive-aggressive computer’s demand that I “ENTER DATA.” In such mundane examples,
pervasively spread across our environment, every detail matters in shaping contextual clarity.

  It bears repeating—especially for digital technology—that there is no such thing as a purely natural interface. Any digital system requires learning (or relying on learned convention) for an artificial user interface of some kind, because there will always be the need to translate the abstraction of digital information into invariants that users can comprehend, whether simple buttons, voice commands, mere labels, or sensor-triggered gestures.

  Figure 13-2. A common workaround for talking with the digital agents among us[265]

  Semantic Function of Simulated Objects

  For most of human existence, an object was either an object or it wasn’t. A picture of an object only represented something; it afforded seeing light of varying shades of color but not taking action with the object depicted.

  In a famous example of this distinction, René Magritte’s painting The Treachery of Images shows a tobacco pipe. Its caption says: “Ceci n’est pas une pipe.” Or, in English, “This is not a pipe.” It’s a popular example for talking about signs, symbols, language, and all manner of fascinating semiotics issues. In our case, it helps make a point about representation in digital systems.