The Economics of Artificial Intelligence
Page 91
grom, and Al Roth (Roth 2002), and even “economist as plumber” (Dufl o
2017) will move beyond the fi elds of market design and development. As
digitization spreads across application areas and sectors of the economy, it
will bring opportunities for economists to develop and implement policies
that can be delivered digitially. Farming advice, online education, health
information and information, government- service provision, government
collections, and personalized resource allocation—all of these create oppor-
tunities for economists to propose policies, design the delivery and imple-
mentation of the policy including randomization or staggered roll- outs to
enable evaluation, and to remain involved through successive rounds of
incremental improvement for adopted policies. Feedback will come more
quickly and there will be more opportunities to gather data, adapt, and
adjust. Economists will be involved in improving operational effi
ciency of
government and industry, reducing costs, and improving outcomes.
Machine- learning methods, when deployed in practice in industry, gov-
ernment, education, and health, lend themselves to incremental improve-
ment. Standard practice in the technology industry is to evaluate incre-
mental improvements through randomized controlled trials. Firms like
Google and Facebook do 10,000 or more randomized controlled trials
of incremental improvements to ML algorithms every year. An emerging
trend is to build the experimentation right into the algorithm using bandit
techniques. As described in more detail earlier, multiarmed bandit is a term
for an algorithm that balances exploration and learning against exploiting
information that is already available about which alternative treatment is
best. Bandits can be dramatically faster than standard randomized con-
trolled experiments (see, e.g., the description of bandits on Google’s web
540 Susan Athey
site: https:// support.google .com/ analytics/ answer/ 2844870?hl=en) because
they have a diff erent goal: the goal is to learn what the best alternative is,
not to accurately estimate the average outcome for each alternative, as in a
standard randomized controlled trial.
Implementing bandit algorithms requires the statistical analysis to be
embedded in the system that delivers the treatments. For example, a user
might arrive at a web site. Based on the user’s characteristics, a contextual
bandit might randomize among treatment arms in proportion to the cur-
rent best estimate of the probability that each arm is optimal for that user.
The randomization would occur “on the fl y” and thus the software for the
bandit needs to be integrated with the software for delivering the treatments.
This requires a deeper relationship between the analyst and the technology
than a scenario where an analyst analyzes historical data “offl
ine” (that is,
not in real time).
Balancing exploration and exploitation involves fundamental economic
concepts about optimization under limited information and resource con-
straints. Bandits are generally more effi
cient and I predict they will come into
much more widespread use in practice. In turn, that will create opportunities
for social scientists to optimize interventions much more eff ectively, and to
evaluate a large number of possible alternatives faster and with less ineffi
-
ciency. More broadly, statistical analysis will come to be commonly placed
in a longer- term context where information accumulates over time.
Beyond bandits, other themes include combining experimental and obser-
vational data to improve precison of estimates (see, e.g., Peysakhovich and
Lada 2016), and making use of large numbers of related experiments when
drawing conclusions.
Optimizing ML algorithms require an objective or an outcome to opti-
mize for. In an environment with frequent and high- velocity experimenta-
tion, measures of success that can be obtained in a short time frame are
needed. This leads to a substantively challenging problem: what are good
measures that are related to long- term goals, but can be measured in the
short term, and are responsive to interventions? Economists will get involved
in helping defi ne objectives and constructing measures of success that can be
used to evaluate incremental innovation. One area of research that is receiv-
ing renewed attention is the topic of “surrogates,” a name for intermediate
measures that can be used in place of long- term outcomes (see, e.g., Athey
et al. 2016). Economists will also place renewed interest on designing incen-
tives that counterbalance the short- term incentives created by short- term
experimentation.
All of these changes will also aff ect teaching. Anticipating the digital
transformation of industry and government, undergraduate exposure
to programming and data will be much higher than it was ten years ago.
Within ten years, most undergraduates will enter college (and most MBAs
will enter business school) with extensive coding experience obtained from
The Impact of Machine Learning on Economics 541
elementary through high school, summer camps, online education, and
internships. Many will take coding and data analysis in college, viewing
these courses as basic preparation for the workforce. Teaching will need to
change to complement the type of material covered in these other classes.
In the short run, more students may arrive at econometrics classes thinking
about data analysis from the perspective that all problems are prediction or
classifi cation problems. They may have a cookbook full of algorithms, but
little intuition for how to use data to solve real- world problems or answer
business or public policy questions. Yet, such questions are prevalent in the
business world: fi rms want to know the return on investment on advertising
campaigns,2 the impact of changing prices or introducing products, and so
on. Economic education will take on an important role in educating stu-
dents in how to use data to answer questions. Given the unique advantages
economics as a discipline has at these methods and approaches, many of the
newly created data science undergraduate and graduate programs will bring
in economists and other social scientists, creating an increased demand
for teaching from empirical economists and applied econometricians. We
will also see more interdisciplinary majors; Duke and MIT both recently
announced joint degrees between computer science and economics. There
are too many newly created data science master’s programs to mention, but
a key observation is that while early programs most commonly have emerged
from computer science and engineering, I predict that these programs will
over time incorporate more social science, or else adopt and teach social
science empirical methods themselves. Graduates entering the workforce
will need to know basic empirical strategies like diff erence- in-diff erences
that often arise in the business world (e.g., some consumers or areas are
exposed to a treatment and not others, and there
are important seasonality
eff ects to control for).
A fi nal prediction is that we will see a lot more research into the societal
impacts of machine learning. There will be large- scale, very important regu-
latory problems that need to be solved. Regulating the transportation infra-
structure around autonomous vehicles and drones is a key example. These
technologies have the potential to create enormous effi
ciency. Beyond that,
reducing transportation costs substantially eff ectively increases the supply
of land and housing in commuting distance of cities, thus reducing housing
costs for people who commute into cities to provide services for wealthier
people. This type of reduction in housing cost would be very impactful for
the cost of living for people providing services in cities, which could reduce
eff ective inequality (which may otherwise continue to rise). But there are a
plethora of policy issues that need to be addressed, ranging from insurance
2. For example, several large technology companies employ economists with PhDs from
top universities who specialize in evaluating and allocating advertising spend for hundreds of millions of dollars of expenditures; see Lewis and Rao (2015) for a description of some of the challenges involved.
542 Susan Athey
and liability, to safety policy, to data sharing, to fairness, to competition
policy, and many others. Generally, the problem of how regulators approach
algorithms that have enormous public impact is not at all worked out. Are
algorithms regulated on outcomes, or on procedures and processes? How
should regulators handle equilibrium eff ects, for example, if one autono-
mous vehicle system makes a change to its driving algorithms, and how is
that communicated to others? How can we avoid problems that have plagued
personal computer software, where bugs and glitches are common following
updates? How do we deal with the fact that having an algorithm used by 1
percent of cars does not prove it will work when used by 100 percent of cars,
due to interaction eff ects?
Another industry where regulation of ML is already becoming prob-
lematic is fi nancial services. Financial- service regulation traditionally con-
cerned processes, rules, and regulations. There is not currently a framework
for cost- benefi t analysis, or deciding how to test and evaluate algorithms,
and determining an acceptable error rate. For algorithms that might have an
eff ect on the economy, how do we assess systematic risks? These are fruitful
areas for future research as well. And of course, there are crucial questions
about how ML will aff ect the future of work, as ML is used across wider
and wider swaths of the economy.
We will also see experts in the practice of machine learning and AI col-
laborate with diff erent subfi elds of economics in evaluating the impact of
AI and ML on the economy.
Summarizing, I predict that economics will be profoundly transformed
by AI and ML. We will build more robust and better- optimized statistical
models, and we will lead the way in modifying the algorithms to have other
desirable properties, ranging from protection against overfi tting and valid
confi dence intervals, to fairness or nonmanipulability. The kinds of research
we do will change; in particular, a variety of new research areas will open
up, with better measurement, new methods, and diff erent substantive ques-
tions. We will grapple with how to reorganize the research process, which
will have increased fi xed costs and larger- scale research labs, for those who
can fund it. We will change our curriculum and take an important seat at
the table in terms of educating the future workforce with empirical and
data science skills. And, we will have a whole host of new policy problems
created by ML and AI to study, including the issues experienced by parts of
the workforce who need to transition jobs when their old jobs are eliminated
due to automation.
21.6 Conclusions
It is perhaps easier than one might think to make predictions about the
impact of ML on economics, since many of the most profound changes are
The Impact of Machine Learning on Economics 543
well underway. There are exciting and vibrant research areas emerging, and
dozens of applied papers making use of the methods. In short, I believe there
will be an important transformation. At the same time, the automation of
certain aspects of statistical algorithms does not change the need to worry
about the things that economists have always worried about: is a causal
eff ect really identifi ed from the data, are all confounders measured, what
are eff ective strategies for identifying causal eff ects, what considerations are
important to incorporate in a particular applied setting, defi ning outcome
metrics that refl ect overall objectives, constructing valid confi dence inter-
vals, and many others. As ML automates some of the routine tasks of data
analysis, it becomes all the more important for economists to maintain their
expertise at the art of credible and impactful empirical work.
References
Abadie, A., A. Diamond, and J. Hainmueller. 2010. “Synthetic Control Methods for
Comparative Case Studies: Estimating the Eff ect of California’s Tobacco Control
Program.” Journal of the American Statistical Association 105 (490): 493– 505.
Asher, S., D. Nekipelov, P. Novosad, and S. Ryan. 2016. “Classifi cation Trees for
Heterogeneous Moment- Based Models.” NBER Working Paper no. 22976, Cam-
bridge, MA.
Athey, S. 2017. “Beyond Prediction: Using Big Data for Policy Problems.” Science
355 (6324): 483– 85.
Athey, S., M. Bayati, N. Doudchenko, G. Imbens, and K. Khosravi. 2017. “Matrix
Completion Methods for Causal Panel Data Models.” Cornell University Library.
arXiv preprint arXiv:1710.10251.
Athey, S., D. Blei, R. Donnelly, and F. Ruiz. 2017. “Counterfactual Inference for
Consumer Choice across Many Product Categories.” Working paper.
Athey, S., D. M. Blei, R. Donnelly, F. J. Ruiz, and T. Schmidt. 2018. “Estimating
Heterogeneous Consumer Preferences for Restaurants and Travel Time Using
Mobile Location Data.” AEA Papers and Proceedings 108:64– 67.
Athey, S., R. Chetty, G. Imbens, and H. Kang. 2016. “Estimating Treatment Eff ects
Using Multiple Surrogates: The Role of the Surrogate Score and the Surrogate
Index.” Cornell University Library. preprint arXiv:1603.09326.
Athey, S., D. Coey, and J. Levin. 2013. “Set- Asides and Subsidies in Auctions.”
American Economic Journal: Microeconomics 5 (1): 1– 27.
Athey, S., D. Eckles, and G. W. Imbens. 2015. “Exact P-Values for Network Interfer-
ence.” NBER Working Paper no. 21313, Cambridge, MA.
Athey, S., and P. A. Haile. 2007. “Nonparametric Approaches to Auctions.” Hand-
book of Econometrics 6:3847– 965.
Athey. S., and G. Imbens. 2015. “A Measure of Robustness to Misspecifi cation.”
American Economic Review 105 (5): 476– 80.
———. 2016. “Recursive Partitioning for Heterogeneous Causal Eff ects.” Proceed-
ings of the National Academy of Sciences 113 (27): 7353– 60.
&
nbsp; ———. 2017. “The State of Applied Econometrics: Causality and Policy Evalua-
tion.” Journal of Economic Perspectives 31 (2): 3– 32.
544 Susan Athey
Athey, S., G. Imbens, T. Pham, and S. Wager. 2017. “Estimating Average Treat-
ment Eff ects: Supplementary Analyses and Remaining Challenges.” American
Economic Review 107 (5): 278– 81.
Athey, S., G. W. Imbens, and S. Wager. 2016. “Approximate Residual Balancing:
De- Biased Inference of Average Treatment Eff ects in High Dimensions.” Cornell
University Library. preprint arXiv:1604.07125.
Athey, S., J. Levin, and E. Seira. 2011. “Comparing Open and Sealed Bid Auctions:
Evidence from Timber Auctions.” Quarterly Journal of Economics 126 (1): 207– 57.
Athey, S., M. M. Mobius, and J. Pál. 2017. “The Impact of Aggregators on Inter-
net News Consumption.” Working Paper no. 3353, Stanford Graduate School of
Business.
Athey, S., J. Tibshirani, and S. Wager. 2017. “Generalized Random Forests.” Cornell
University Library. https:// arxiv .org/ abs/ 1610.01271.
Athey, S., and S. Wager. 2017. “Effi
cient Policy Estimation.” Cornell University
Library. https:// arxiv .org/ abs/ 1702.02896.
Bai, J., and S. Ng. 2008. “Large Dimensional Factor Analysis.” Foundations and
Trends® in Econometrics 3 (2): 89– 163.
Banbura, M., D. Giannone, M. Modugno, and L. Reichlin. 2013. “Now- Casting
and the Real- Time Data Flow.” ECB Working Paper no. 1564, European Central
Bank.
Bastani, H., and M. Bayati. 2015. “Online Decision- Making with High- Dimensional
Covariates.” SSRN Electronic Journal. https:// www .researchgate .net/ publication
/ 315639905_Online_Decision- Making_with_High- Dimensional_Covariates.
Belloni, A., V. Chernozhukov, and C. Hansen. 2014. “High- Dimensional Methods
and Inference on Structural and Treatment Eff ects.” Journal of Economic Perspec-
tives 28 (2): 29– 50.
Bickel, P. J., C. A. Klaassen, Y. Ritov, J. Klaassen, J. A. Wellner, and Y. Ritov. 1993.
Effi
cient and Adaptive Estimation for Semiparametric Models. Baltimore: Johns
Hopkins University Press.
Bjorkegren, D., and D. Grissen. 2017. “Behavior Revealed in Mobile Phone Usage
Predicts Loan Repayment.” SSRN Electronic Journal. https:// www .research