social and political power to promulgate rules (for example, governments
and courts); and as the socially legitimized constructions that instantiate
rules and through which economic agents act (for example, fiat money and
markets). In whichever sense institutions are construed, the three authors
agree that they cannot be adequately understood from a purely economic,
purely political, or purely social point of view. Economics, politics, and society are inextricably mixed in the processes whereby institutions come into
being. And they change and determine economic, political, and social action.
North also insists that institutions have a cognitive dimension through the
aggregate-level “belief systems” that sustain them and determine the direc-
tions in which they change.
North takes up the question of the emergence of institutions from a func-
tionalist perspective: institutions are brought into being “in order to reduce uncertainty,” that is, to make agents’ worlds predictable enough to afford rec-ognizable opportunities for effective action. In particular, modern economies
depend upon institutions that provide low transaction costs in impersonal
markets.
[ 100 ] Complexity and the Economy
Shubik takes a different approach. His analysis starts from his notion of strategic market games. These are “fully defined process models” that specify
actions “for all points in the set of feasible outcomes.” He shows how, in the context of constructing a strategic market game for an exchange economy
using fiat money, the full specification requirement leads to the logical necessity of certain kinds of rules that Shubik identifies with financial institutions.
Geanakoplos’ paper makes a similar point to Shubik’s. Financial instruments
represent promises, he argues. What happens if someone cannot or will not
honor a promise? Shubik already introduced the logical necessity of one
institution, bankruptcy law, to deal with defaults. Geanakoplos introduces
another, collateral. He shows that, in equilibrium, collateral as an institution has institutional implications—missing markets.
Finally, in his note concluding the volume, Philip Anderson provides a
physicist’s perspective on a point that Fernand Braudel argues is a central
lesson from the history of long-term socioeconomic change. Averages and
assumptions of agent homogeneity can be very deceptive in complex sys-
tems. And processes of change are generally driven by the inhabitants of the
extreme tails of some relevant distribution. Hence, an interesting theoretical question from the Santa Fe perspective is: How do distributions with extreme
tails arise, and why are they so ubiquitous and so important?
WHAT COUNTS AS A PROBLEM AND AS A SOLUTION. While the papers
here have much to say on cognition and structure, they contain much less
discussion on what constitutes a problem and solution from this new view-
point. Perhaps this is because it is premature to talk about methods for gen-
erating and assessing understanding when what is to be understood is still
under discussion. While a few of the papers completely avoid mathemat-
ics, most of the papers do present mathematical models—whether based
on statistical mechanics, strategic market games, random graphs, popula-
tion games, stochastic dynamics, or agent-based computations. Yet some-
times the mathematical models the authors use leave important questions
unanswered. For example, in what way do equilibrium calculations provide
insight into emergence? This troublesome question is not addressed in any
of the papers, even those in which models are presented from which equilib-
ria are calculated—and insight into emergence is claimed to result. Blume
raises two related issues in his discussion of population games: whether the
asymptotic equilibrium selection theorems featured in the theory happen
“soon enough” to be economically interesting; and whether the invariance
of the “global environment” determined by the game and interaction model
is compatible with an underlying economic reality in which rules of the
game undergo endogenous change. It will not be easy to resolve the inher-
ent tension between traditional mathematical tools and phenomena that
may exhibit perpetual novelty.
Proce ss and emergence in t He economy [ 101 ]
As we mentioned previously, several of the papers introduce less traditional, agent-based models. Kollman, Miller, and Page discuss both advantages and
difficulties associated with this set of techniques. They end up expressing cautious optimism about their future usefulness. Tesfatsion casts her own paper
as an illustration of what she calls “the alife approach for economics, as well as the hurdles that remain to be cleared.” Perhaps the best recommendation we
can make to the reader with respect to the epistemological problems associ-
ated with the process-and-emergence perspective is simple. Read the papers,
and see what you find convincing.
REFERENCES
1. Baker, W. “The Social Structure of a National Securities Market.” Amer. J. Sociol.
89, (1984): 775–811.
2. Clark, A. Being There: Putting Brain, Body, and World Together Again. Cambridge, MA: MIT Press, 1997.
3. Eigen, M., and P. Schuster. The Hypercycle. Berlin: Springer Verlag, 1979.
4. Fontana, W., and L. Buss. “The Arrival of the Fittest: Toward a Theory of
Biological Organization.” Bull. Math. Biol. 56 (1994): 1–64.
5. Holland, J. H. “The Global Economy as an Adaptive Process.” In The Economy as an Evolving Complex System, edited by P. W. Anderson, K. J. Arrow, and D. Pines, 117–124. Santa Fe Institute Studies in the Sciences of Complexity, Proc. Vol. V.
Redwood City, CA: Addison-Wesley, 1988.
6. Hutchins, E. Cognition in the Wild. Cambridge, MA: MIT Press, 1995.
7. Noria, N., and R. Eccles (Eds.) Networks and Organizations: Structure, Form, and Action. Cambridge, MA: Harvard Business School Press, 1992.
8. Wasserman, W., and K. Faust. Social Network Analysis: Methods and Applications.
Cambridge, UK: Cambridge University Press, 1994.
[ 102 ] Complexity and the Economy
CHAPTER 6
All Systems Will Be Gamed
Exploitive Behavior in Economic and Social Systems
W. BRIAN ARTHUR1
After the 2008 Wall Street crash, it became clear to many economists that financial systems, along with other social and economic systems, were not immune to being manipulated by small groups of players to their own advantage. And so, two natural questions arose. For a given policy design or proposed economic system, could such manipulation be foreseen in advance and possibly prevented? And could we design methods—possibly automatic ones—that would test proposed policy systems for possible failure modes and for their vulnerability to possible manipulation, and thereby prevent such behavior in the future?
The paper argues that exploitive behavior within the economy is not rare and falls into specific classes; that policy studies can be readily extended to investigate the possibility of the policy’s being “gamed;” and that economics needs a strong sub-discipline of failure-mode analysis, parallel to the successful failure-mode-analysis disciplines within structural engineering and aircraft design. The paper was written in 2010 when I was with IBM Almaden’s Smarter Planet Platform for Analysis and Simulation of Health (SPLASH) group. It is published here for the first time.1
There is a general rule in social and economic life: given any syste
m, people
will find a way to exploit it. Or to say this more succinctly: All systems
will be gamed. I do not mean to be cynical here. Rather, I am making the
general observation that given any governmental system, any legal system,
1. I thank my fellow IBM team members, Paul Maglio, Peter Haas, and Pat Selinger for useful comments; and also Sai Hung Cheung and Daria Rothmaier.
regulatory system, corporate system, election system, set of policies, set of organizational rules, set of international agreements, people can—and will—
find unexpected ways to exploit it to their advantage. “Show me a 50-foot
wall,” said Arizona’s governor Janet Napolitano, speaking in 2005 of illegal
immigration at the US-Mexico border, “and I’ll show you a 51-foot ladder.”
Foreseeing 51-foot ladders may not be particularly challenging—
Napolitano is merely making a wry political point. But anticipating more
generally how exploitive behavior can arise in a given policy system is challenging; there are many ways in which systems can be exploited and some
are by no means obvious. Yet we do need to foresee possible manipulations,
not least because they can sometimes have disastrous consequences. Consider
the aftermath of Russia’s 1990 transition from planned socialism to capital-
ism, in which a small number of well-positioned players seized control of the
state’s newly freed assets. Or consider California’s 2000 freeing of its energy market, in which a small number of suppliers were able to manipulate the
market to the detriment of the state. Or consider Iceland’s banking system
in 2008, where a few financial players who had taken control of the state’s
banks used depositors’ assets to speculate in overseas property markets and
ran the banks into insolvency. Or consider Wall Street’s loosely regulated
mortgage-backed securities market in 2008, in which easy credit and compli-
cated derivative products built a highly unstable structure that spectacularly collapsed. All these systems were manipulated—some were “gamed,” to use a
stronger term. All, in retrospect, posed incentives that rendered them open to manipulation—and all careened into eventual system breakdowns.
This raises an obvious question. Given that economics is sophisticated and
that economists study proposed policy systems in advance, how could these
various economic disasters have happened? In the cases I mentioned, some
economists did indeed foresee possibilities for future exploitation and warn
of these. But such warnings normally have little effect. The reason is that economics, in the way it is practiced, contains a bias that inhibits economists from seeing future potential exploitation. Economic analysis assumes equilibrium
of the system in question, and by definition equilibrium is a condition where
no agent has any incentive to diverge from its present behavior. It follows that for any system being studied invasive or exploitive behavior cannot happen: If a system could be invaded, some agents would be initiating new behavior, and
the system could not have been in equilibrium. Equilibrium economics then,
by its base assumptions, is not primed to look for the exploitation of systems, and as a result systematic studies of how systems might fail or be exploited are not central to how the discipline thinks.2
2. For critiques of economics in the face of the 2008 financial crisis and other crises, see Colander et al., 2008; and Koppl and Luther, 2010.
[ 104 ] Complexity and the Economy
In this paper I want to get away from the equilibrium assumption and take as our basis a different, nonequilibrium assumption: that any policy system at any time presents incentives to the parties engaged in it, and these incentives may in turn induce parties to discover ways in which they might privately
benefit that policy designers had not thought of. Given this, we would want
to know how exploitive behavior for policy systems might typically arise, and
how we can use formal modeling and analysis to allow for such behavior, and
to foresee or even warn of it in advance.
I will pose our problem of foreseeing possible exploitation as four ques-
tions I will look at in sequence. First, what are the causes of exploitive behavior and how does it typically arise? Second, given a particular economic system or proposed policy, how might we anticipate where it might fail, and what can
we learn from disciplines such as structural engineering that try to foresee
potential failure modes, and could help us in this? Third, how can we construct models of systems being gamed or exploited, and of agents in these models
“discovering” ways to exploit such systems? And fourth, what are the future
prospects for constructing artificially intelligent methods that could automatically anticipate how economic and social systems might be exploited? Fully
definitive answers to these questions are of course not possible, but I hope the discussion here will at least open the subject for debate.
Before we go on, a word about some of the terms I will use. Exploitation has two meanings: “to use something in order to gain a benefit,” and to take “selfish or unfair advantage of a person or situation, usually for personal gain.”3
The first meaning suits us well (note it is not necessarily pejorative), but the second also covers many of the cases I will talk about. Gaming itself has a more pernicious meaning: it denotes people using a system cynically to their own
ends, often in a way that betrays trust placed in them and harms other peo-
ple.4 I will also talk of policy systems, meaning economic or social or military or business or governmental systems that play out over time, given a set of
policies that define them. The 2010 Obama Affordable Health Care system is
a policy system.
3. Microsoft Word Dictionary (1991) uses “exploitation” in its first sense, as the use and refinement of existing opportunities, and contrasts this with “exploration,” the ongoing search for new opportunities. See March (1991) on this. “Exploitation” in this paper contains elements of both of these: we are talking of agents exploring for opportunities to exploit.
4. Wikipedia (October 9, 2010) defines gaming as “[using] the rules and procedures meant to protect a system in order, instead, to manipulate the system for a desired outcome.”
all syst ems Will Be g amed [ 105 ]
CAUSES OF EXPLOITIVE BEHAVIOR
Before we talk about modeling exploitive behavior, it will be useful to build up some knowledge about its causes and mechanisms.
Our first observation is that exploitive behavior is not rare. This is not
because of some inherent human tendency toward selfish behavior; it is because all policy systems—all social policies—pose incentives that are reacted to by
groups of agents acting in their own interest, and often these reactions are
unexpected and act counter to the policy’s intentions. Examples are legion.
The 2003 US invasion of Iraq—a military policy system—was well planned
and well executed, but it generated insurgency, a less than fully expected reaction to the presence of American soldiers that went on to obstruct US goals
in Iraq. The 1965 Medicare system, launched under Lyndon Johnson with the
purpose of providing health care for the elderly, paid fee-for-service, compensating hospitals and physicians for their incurred costs of treatment. Hospitals and physicians in the program responded by purchasing expensive equipment
and providing services that were unnecessary. As a result, within five years of its inception, the program’s costs nearly tripled (Mahar, 2006). A decade or
two later, the United States opened health care to market f
orces. The freeing of the market was intended to produce competition and to lower costs. Instead it
produced a system where each of the key players found specific ways to work
the system to their own advantage, to the detriment of the system as a whole.
Maher (2006) describes the outcome as “a Hobbesian marketplace” that pitted
“the health care industry’s players against one another: hospital vs. hospital, doctor vs. hospital, doctor vs. doctor, hospital vs. insurer, insurer vs. hospital, insurer vs. insurer, insurer vs. drugmaker, drugmaker vs. drugmaker.”
These examples are large-scale ones, but exploitation happens on every
scale. Apartment building managers have been known to visit their competi-
tors’ buildings and post negative ratings online to enhance their own com-
petitive standing. Whatever the scale at which exploitation takes place, its
frequency of occurrence should give us pause about implementing any social
policy without thinking through how it could potentially be used to players’
advantage, and it should also caution us about accepting the results of eco-
nomic models designed to demonstrate a policy system’s outcome. In fact,
it should caution us about accepting the results of all policy models without questioning their built-in assumptions.
But just how should we question the outcome of policy systems? The exam-
ples I have given seem scattered and unique, so it doesn’t seem easy to build
general insights from them. It would be better if we could find generic categories of exploitation, standard hacks, or patterns of behavior or incentives that we see repeated from one circumstance to another. Or to put this another way,
it would be useful if we had a “failure mode analysis” tradition in economics
for assessing policy systems. Such a tradition exists in other disciplines where
[ 106 ] Complexity and the Economy
life or safety or well-being are at stake: Failure mode analysis in engineering investigates the ways in which structures have failed in the past and might
fail or not function as intended; preventive medicine and disease control
investigates the causes of diseases, death, and epidemics and looks to their
Complexity and the Economy Page 20