by Finn Brunton
A U S E R ’ S G U I D E
FOR PRIVACY AND PROTEST
Finn Brunton Helen Nissenbaum
The MIT Press
Cambridge, Massachusetts
London, England
© 2015 Finn Brunton and Helen Nissenbaum
All rights reserved. No part of this book may be reproduced in any form by any electronic or mechanical means (including photocopying, recording, or information storage and retrieval) without permission in writing from the publisher.
MIT Press books may be purchased at special quantity discounts for business or sales promotional use. For information, email [email protected].
Set in PF Din Text Cond Pro by Toppan Best-set Premedia Limited. Printed and bound in the United States of America.
Library of Congress Cataloging-in-Publication Data are available.
ISBN: 978-0-262-02973-5
10 9 8 7 6 5 4 3 2 1
CONTENTS
Acknowledgments ix
Introduction 1
I AN OBFUSCATION VOCABULARY
1 Core Cases 8
1.1 Chaff: defeating military radar 8
1.2 Twitter bots: filling a channel with noise 9
1.3 CacheCloak: location services without location tracking 12
1.4 TrackMeNot: blending genuine and artificial search queries 13
1.5 Uploads to leak sites: burying significant files 14
1.6 False tells: making patterns to trick a trained observer 15
1.7 Group identity: many people under one name 15
1.8 Identical confederates and objects: many people in one outfit 16
1.9 Excessive documentation: making analysis inefficient 17
1.10 Shuffling SIM cards: rendering mobile targeting uncertain 18
1.11 Tor relays: requests on behalf of others that conceal personal
traffic 19
1.12 Babble tapes: hiding speech in speech 21
1.13 Operation Vula: obfuscation in the struggle against Apartheid 21
2 Other Examples 25
2.1 Orb-weaving spiders: obfuscating animals 25
2.2 False orders: using obfuscation to attack rival businesses 25
2.3 French decoy radar emplacements: defeating radar detectors 26
2.4 AdNauseam: clicking all the ads 26
2.5 Quote stuffing: confusing algorithmic trading strategies 27
2.6 Swapping loyalty cards to interfere with analysis of shopping
patterns 28
2.7 BitTorrent Hydra: using fake requests to deter collection of
addresses 29
2.8 Deliberately vague language: obfuscating agency 30
2.9 Obfuscation of anonymous text: stopping stylometric analysis 31
2.10 Code obfuscation: baffling humans but not machines 33
2.11 Personal disinformation: strategies for individual disappearance 35
2.12 Apple’s “cloning service” patent: polluting electronic profiling 36
2.13 Vortex: cookie obfuscation as game and marketplace 37
2.14 “Bayesian flooding” and “unselling” the value of online identity 38
2.15 FaceCloak: concealing the work of concealment 39
2.16 Obfuscated likefarming: concealing indications of manipulation 40
2.17 URME surveillance: “identity prosthetics” expressing protest 40
2.18 Manufacturing conflicting evidence: confounding investigation 41
II UNDERSTANDING OBFUSCATION
3 Why Is Obfuscation Necessary? 45
3.1 Obfuscation in brief 45
3.2 Understanding information asymmetry: knowledge and power 48
3.3 The fantasy of opting out 53
3.4 Weapons of the weak: what obfuscation can do 55
3.5 Distinguishing obfuscation from strong privacy systems 58
4 Is Obfuscation Justified? 63
4.1 Ethics of obfuscation 64
4.2 From ethics to politics 70
vi
CONTENTS
5 Will Obfuscation Work? 84
5.1 Obfuscation is about goals 85
5.2 I want to use obfuscation … 87
… to buy some time 88
… to provide cover 88
… for deniability 89
… to prevent individual exposure 89
… to interfere with profiling 90
… to express protest 90
5.3 Is my obfuscation project … 90
… individual, or collective? 91
… known, or unknown? 92
… selective, or general? 92
… short-term, or long-term? 94
Epilogue 97
Notes 99
Bibliography 113
Index 121
CONTENTS
vii
ACKNOWLEDGMENTS
This book began with technology—the TrackMeNot project—and we owe
our deepest thanks to Daniel Howe, who got it on its feet, and Vincent
Toubiana, who joined the effort, expanded its scope, and continues tirelessly to support it and its adopters. Feedback and comments from the community of users and from the privacy community at large, and a joint technical paper with Lakshminarayanan Subramanian (Vincent Toubiana, Lakshminarayanan
Subramanian, and Helen Nissenbaum, “TrackMeNot: Enhancing the Privacy of
Web Search”) have opened our eyes to its potential and limitations. More
recently, creating and launching a second system, AdNauseam, with Daniel
Howe, in collaboration with the designer Mushon Zer-Aviv, further expanded our perspective on obfuscation and on the need for a deeper, systematic
appreciation of what it offers as a method and a strategy.
As we began to work on a general understanding of obfuscation, we were
able to explore many of the concepts in a paper published in First Monday and a chapter in Privacy, Due Process and the Computational Turn, which benefited enormously from review and editorial feedback in those venues.
Obfuscation became a book with the encouragement and thorough advice of the reviewers and of Marguerite Avery, Gita Manaktala, Susan Buckley,
Katie Helke, and Paul Bethge at the MIT Press. Our thanks to all of them. Emily Goldsher-Diamond did meticulous work as a research assistant, as well as
organizing many other aspects of this project. Work on this book through all its drafts was supported by grants from the National Science Foundation (ITR-0331542: Sensitive Information in a Wired World), from EAGER (CNS-1355398: Values in Design for Future Internet Architecture—Next Phase), from the Air Force Office of Scientific Research (MURI-ONR BAA 07-036: Collaborative
Policies and Assured Information Sharing), and from the Intel Science and
Technology Center for Social Computing. Support from these grants provided time, technology, and a collegial context for pursuing this project and bringing it to fruition.
Two major events helped shape and refine our thinking about obfuscation.
One was the Symposium on Obfuscation (February 15, 2014), jointly organized by New York University’s Department of Media, Culture, and Communication
and the Information Law Institute and co-sponsored by the Intel Science and Technology Center for Social Computing. For making this event possible, we would like to thank Nicole Arzt, Emily Goldsher-Diamond, Dove Helena
Pedlosky, Melissa Lucas-Ludwig, Erica Robles-Anderson, and Jamie Schuler—
and, above all, Seda Gürses, who organized, structured, and shaped so much of the day. Every single speaker had a direct effect on our manuscript. The other event was the ongoing conversation of the Privacy Research Group at
NYU, at whose weekly seminars we presented several stages of this material.
Other opportunities to present and test aspects of this work have been
enormously productive, and our ideas have been greatly improved by the
responses of supporters, critics, believers, and skeptics. These opportunities have included a joint MIT Center for Civic Media and Comparative Media Studies Colloquium; The New School for Social Research 2014 Graduate Conference;
New Media Salon, Tel Aviv; Communications and Journalism Departmental
Seminar, Hebrew University of Jerusalem; IBM R&D Labs, Haifa; Eyebeam Art
+ Technology Center; HotPETS 2013; Computers, Privacy and Data Protection, Brussels; and the Surveillance Studies Conference, Queens University.
We are deeply grateful for friends and colleagues with whom we could
discuss obfuscation as it developed, and who offered feedback, criticism,
encouragement, and ideas. In particular we would like to thank Julia Angwin, Solon Barocas, danah boyd, Claudia Diaz, Cynthia Dwork, Cathy Dwyer,
Tarleton Gillespie, Mireille Hildebrandt, Ari Juels, Nick Montfort, Deirdre Mulligan, Arvind Narayanan, Martijn van Otterloo, Ira Rubinstein, Ian Spiro, Luke Stark, Katherine Strandburg, Matthew Tierney, Joe Turow, Janet Vertesi, Tal Zarsky, Malte Ziewitz, and Ethan Zuckerman.
Finally, this book would not have been possible without the support of our professional home base, the Department of Media, Culture, and Communication at New York University. Thanks to you all!
x
ACKNOWLEDGMENTS
INTRODUCTION
We mean to start a revolution with this book. But not a big revolution—at
least, not at first. Our revolution does not rely on sweeping reforms, on a comprehensive Year Zero reinvention of society, or on the seamless and perfectly uniform adoption of a new technology. It is built on preexisting components—
what a philosopher would call tools ready-to-hand, what an engineer would
call commodity hardware—that are available in everyday life, in movies, in software, in murder mysteries, and even in the animal kingdom. Although its lexicon of methods can be, and has been, taken up by tyrants, authoritarians, and secret police, our revolution is especially suited for use by the small players, the humble, the stuck, those not in a position to decline or opt out or exert control over our data emanations. The focus of our limited revolution is on mitigating and defeating present-day digital surveillance. We will add
concepts and techniques to the existing and expanding toolkit for evasion, noncompliance, outright refusal, deliberate sabotage, and use according to our terms of service. Depending on the adversary, the goals, and the resources, we provide methods for disappearance, for time-wasting and analysis-frustrating, for prankish disobedience, for collective protest, for acts of individual redress both great and small. We draw an outline around a whole
domain of both established and emerging instances that share a common
approach we can generalize and build into policies, software, and action. This outline is the banner under which our big little revolution rides, and the space it defines is called obfuscation.
In a sentence: Obfuscation is the deliberate addition of ambiguous,
confusing, or misleading information to interfere with surveillance and data collection. It’s a simple thing with many different, complex applications and uses. If you are a software developer or designer, obfuscation you build into your software can keep user data safe—even from yourself, or from whoever
acquires your startup—while you provide social networking, geolocation, or other services requiring collection and use of personal information. Obfuscation also offers ways for government agencies to accomplish many of the
goals of data collection while minimizing the potential misuses. And if you are a person or a group wanting to live in the modern world without being a subject of pervasive digital surveillance (and an object of subsequent analysis),
obfuscation is a lexicon of ways to put some sand in the gears, to buy time, and to hide in the crowd of signals. This book provides a starting point.
Our project has tracked interesting similarities across very different domains in which those who are obliged to be visible, readable, or audible have
responded by burying salient signals in clouds and layers of misleading
signals. Fascinated by the diverse contexts in which actors reach for a strategy of obfuscation, we have presented, in chapters 1 and 2, dozens of detailed instances that share this general, common thread. Those two chapters, which make up part I of the book, provide a guide to the diverse forms and formats that obfuscation has taken and demonstrate how these instances are crafted and implemented to suit their respective goals and adversaries. Whether on a social network, at a poker table, or in the skies during the Second World War, and whether confronting an adversary in the form of a facial-recognition
system, the Apartheid government of 1980s South Africa, or an opponent
across the table, properly deployed obfuscation can aid in the protection of privacy and in the defeat of data collection, observation, and analysis. The sheer range of situations and uses discussed in chapters 1 and 2 is an inspira-tion and a spur: What kind of work can obfuscation do for you?
The cases presented in chapter 1 are organized into a narrative that introduces fundamental questions about obfuscation and describes important
approaches to it that are then explored and debated in part II of the book. In chapter 2, shorter cases illustrate the range and the variety of obfuscation applications while also reinforcing underlying concepts.
Chapters 3–5 enrich the reader’s understanding of obfuscation by consid-
ering why obfuscation has a role to play in various forms of privacy work; the ethical, social, and political problems raised by using obfuscatory tactics; and ways of assessing whether obfuscation works, or can work, in particular scenarios. Assessing whether an obfuscation approach works entails under-
standing what makes obfuscation distinct from other tools and understanding its particular weaknesses and strengths. The titles of chapters 3–5 are framed as questions.
The first question, asked in chapter 3, is “Why is obfuscation necessary?”
In answering that question, we explain how the challenges of present-day
digital privacy can be met by obfuscation’s utility. We point out how obfuscation may serve to counteract information asymmetry, which occurs when data 2
INTRODUCTION
about us are collected in circumstances we may not understand, for purposes we may not understand, and are used in ways we may not understand. Our
data will be shared, bought, sold, managed, analyzed, and applied, all of which will have consequences for our lives. Will you get a loan, or an apartment, for which you applied? How much of an insurance risk or a credit risk are you?
What guides the advertising you receive? How do so many companies and
services know that you’re pregnant, or struggling with an addiction, or plan-ning to change jobs? Why do different cohorts, different populations, and different neighborhoods receive different allocations of resources? Are you going to be, as the sinister phrase of our current moment of data-driven anti-terrorism has it, “on a list”? Even innocuous or seemingly benign work in this domain has consequences worth considering. Obfuscation has a role to play, not as a replacement for governance, business conduct, or technological interventions, or as a one-size-fits-all solution (again, it’s a deliberately small, distributed revolution), but as a tool that fits into the larger network of privacy practices. In particular, it’s a tool particularly well suited to the category of people without access to other modes of recourse, whether at a particular
moment or in general—people who, as it happens, may be unable to deploy
optimally configured privacy-protection tools because they are on the weak side of a particular i
nformation-power relationship.
Similarly, context shapes the ethical and political questions around
obfuscation. Obfuscation’s use in multiple domains, from social policy to social networks to personal activity, raises serious concerns. In chapter 4, we ask “Is obfuscation justified?” Aren’t we encouraging people to lie, to be willfully inaccurate, or to “pollute” with potentially dangerous noise databases that have commercial and civic applications? Aren’t obfuscators who use commercial
services free riding on the good will of honest users who are paying for
targeted advertising (and the services) by making data about themselves
available? And if these practices become widespread, aren’t we going to be collectively wasting processing power and bandwidth? In chapter 4 we address these challenges and describe the moral and political calculus according to which particular instances of obfuscation may be evaluated and found to be acceptable or unacceptable.
What obfuscation can and can’t accomplish is the focus of chapter 5. In
comparison with cryptography, obfuscation may be seen contingent, even
shaky. With cryptography, precise degrees of security against brute-force
INTRODUCTION
3
attacks can be calculated with reference to such factors as key length, processing power, and time. With obfuscation such precision is rarely possible, because its strength as a practical tool depends on what users want to accomplish and on what specific barriers they may face in respective circumstances of use. Yet complexity does not mean chaos, and success still rests on careful attention to systematic interdependencies. In chapter 5 we identify six common goals for an obfuscation project and relate them to design dimensions. The goals include buying some time, providing cover, deniability, evading observation, interfering with profiling, and expressing protest. The aspects of design we identify include whether an obfuscation project is individual or collective, whether it is known or unknown, whether it is selective or general, and
whether it is short-term or long-term. For some goals, for instance, obfuscation may not succeed if the adversary knows that it is being employed; for other goals—such as collective protest or interference with probable cause and production of plausible deniability—it is better if the adversary knows that the data have been poisoned. All of this, of course, depends on what resources are available to the adversary—that is, how much time, energy, attention, and money the adversary is willing to spend on identifying and weeding out obfuscating information. The logic of these relationships holds promise because it suggests that we can learn from reasoning about specific cases how to