Trillion Dollar Economists_How Economists and Their Ideas have Transformed Business

by Robert Litan

  23. For a great description of this enterprise, see Hank Adams, “Why It’s Never Been More Fun to Watch Sports,” The Atlantic, October 2013, 18–20.

  24. For more details on the subjects discussed in this section, see Cass Sunstein, Simpler: The Future of Government (New York: Simon & Schuster, 2103).

  Chapter 6

  Experiments in Economics and Business

  Economics, at least as it was taught to me, was a non-experimental social science. Like others in this category—sociology, anthropology, or political science—economists could not, it was thought, conduct experiments and test hypotheses like their counterparts in the hard sciences. The chemist, the physicist, or medical researchers can control for all factors other than the one hypothesis he or she wants tested. The economist could do no such thing, but instead had to figure out how an economy already working actually worked. Or, as one common joke has put it, the economist’s job is to explain how something that works in practice works in theory.

  Business pretty much worked this way, too, and for many firms it still does. Individuals or entrepreneurs invent stuff and then try to sell it. The farsighted and strong-willed ones are successful. Henry Ford knew what kind of car the masses of Americans would buy, and what his Ford Motor company would make: a simple black Model T. If consumers didn’t like it, they could go elsewhere. Likewise, Steve Jobs was famous for knowing what electronic devices consumers wanted and what they wanted the designs to look like. And he was mostly right. There are countless other examples of other companies and entrepreneurs who behave with such self-confidence and have been rewarded with success.

  This chapter is about a very different way of thinking—the use of experiments in both economics and business, that has affected both domains but in parallel fashion. So far, there has been essentially no cross-pollination of lessons learned in the two arenas.

  Nonetheless, I devote an entire chapter in this book to experimentation, for multiple reasons. First, experimentalism is the way much, if not most, of the world operates. It is not just scientists who first formulate hypotheses and test them, but also empirically oriented economists, which is most economists these days, who go about their business in the same fashion, although not in precisely the way that scientists do. Increasingly, many businesses are also borrowing scientific testing principles before and during the introduction of new products and services, as well as refining those they already are selling.

  Second, experimental techniques are on the cutting edge of both economics and the business world and for that reason alone they deserve attention.

  Third, my idealistic hope is that practitioners from each domain will begin to learn more from each other and, in particular, that some of those reading this book will not only draw that conclusion but actually apply it in their daily endeavors.

  In what follows, I distinguish between two types of experiments. In economics, I first speak of laboratory experiments, which have their analogue in focus groups in the business world, since each type of experiment is conducted outside of the real world in an effort to improve the accuracy of predicting how those who live and work in that world will behave or react. I then turn my attention to field experiments, or those conducted with real-world subjects: randomized controlled treatments (RCTs) in economics (borrowed from pharmaceutical testing), and A/B testing and variations thereof in the business world. I close the chapter by discussing the importance of experimentation in innovation and entrepreneurship, which drives economic growth and thus, in a sense, blends the two separate worlds of academe and business I discuss in the chapter.

  Economics in the Lab: Vernon Smith and Experimental Economics

  The notion that economists could learn something from a lab-like setting was so heretical that it took an iconoclastic economist from my hometown, Wichita, Kansas, to begin changing the way economists think and the way at least some economics research is now done.

  That individual, Vernon Smith, did what came naturally not only to hard physical scientists, but to many businesspeople every day: He began to experiment, initially with students in his classrooms at the various universities where he has taught, and later in more controlled settings with students or other young people outside the classroom.

  Unlike most other economists, who we know about only through their own writings or in some cases through essays or biographies written about them by others, we know about Smith both through his professional work and his remarkably candid autobiography.1 I draw heavily on that work in summarizing Smith and his ideas here. One other note: Unbeknownst to me as I was drafting this part of the book, Bloomberg View columnist Megan McCardle was finishing up her own terrific book, The Upside of Down, which also discusses Smith’s work in detail. I recommend her book to you for a lot of reasons, but if you want to know more about Smith and his experiments, you’ll find it there.2

  Smith devoted much of his professional life to experimental economics by accident, mostly through his teaching, where he quickly became unsatisfied with the conventional material that was then and still is found in most economics courses. He credits in his autobiography the university that first hired him after graduate school, Purdue, and the chairman of its economics department in the 1950s and 1960s, Em Weiler, with nurturing his interest in experiments as a way of demonstrating to students how Smith believed economies, and more importantly the actors within them, really behaved. He refined his notions about experimental economics at numerous other universities, but mostly at the University of Arizona.

  Drawing on the pioneering work of Harvard economist Edward Chamberlin, who used mock auctions to demonstrate how supply and demand curves work, Smith greatly expanded the use of experiments to confirm basic economic propositions. Like Chamberlin, Smith used students in classroom settings to make bids and offers, illustrating in the process how prices and quantities of a hypothetical commodity converge to their theoretical competitive equilibrium. He found in the course of running these experiments that it doesn’t take a large number of competitors to generate the competitive result found in the supply and demand graphs that populate introductory economics textbooks.

  Vernon Smith: Iconoclast

  Smith’s background is unlike most other economists I have known or those you have read about in this book, and almost surely contributed to his skeptical, questioning approach to economic research.

  Smith’s early life, though it predated the Depression, sounds a lot more like The Grapes of Wrath than the relatively sheltered academic lives of the typical economist, whose fathers, and in some cases mothers, taught at a university, or at least a high school, and thus had homes where one was expected to grow up and pursue an intellectual career. Smith’s father worked for a railroad and did not earn much. As a result, Smith grew up relatively poor, in a household where intellectual discussions were rare.

  Reading his autobiography, it was amazing to me how from his essentially hardscrabble background Smith found his way from North High School and Friends University in Wichita, Kansas, to the California Institute of Technology for his undergraduate degree, and later to Harvard University for his PhD (he switched to economics along the way at the University of Kansas, which he attended as a graduate student after college).

  Smith began his varied academic career at the Krannert School of Management at Purdue, but moved through many other universities, staying the longest (26 years) at the University of Arizona, where he conducted much of the experimental work that eventually earned him a Nobel Prize. Smith has also taught at Stanford, Brown, George Mason, and, most recently, Chapman University in southern California, which gave him the resources to open an experimental economics lab using local area high school students, among others, as subjects.

  During the course of his career Smith has published on a broad array of topics outside experimental economics, including finance and natural resources. Almost unique among academic economists, Smith shared his byline on a number of his important papers with his undergraduate student
s at the University of Arizona who helped him design and computerize a number of his experiments.3

  Smith worked with various colleagues over the years to refine his experiments, and to show they could be applied outside the classroom and in the real world. In his words, Smith confirmed to me that by far the most practical application of his ideas was in the design of bidding procedures for wholesale electric power, which he developed in conjunction with Stephen Rassenti at Arizona. Initially, Smith and Rassenti proposed such a system for the power commission in the state, which rejected the ideas as too impractical. Ironically, they were picked up and implemented in the 1990s in New Zealand and Australia, and also by the electric utility company Ohio Edison.4

  It is difficult to overstate the extent to which Smith’s work was long viewed to be out of the mainstream by the rest of the profession, where members spent their time either theorizing or using regression analysis and other statistical techniques (in both cases, using increasingly sophisticated mathematics) to test those theories. The standard view is that economic behavior could only be discerned when people or firms were using their own money in real-world settings. The notion that lessons could be drawn from laboratory settings where individuals were given play (or even small amounts of real) money and then tasked with spending it or using it in some fashion for purposes designed by the experimenters was not only the exception rather than the rule, but looked down on by many mainstream economists––until Smith was awarded the Nobel Prize, of course.

  As evidence, consider the fact that fully 8 percent of all the papers published in several leading economics journals in 2011 used experimental methods or fell into the experimental economics category.5 That figure was up from essentially zero in 1973, the first year covered by the study. That same study also showed a sharp upward trend in empirical papers published with the author’s own data set, and downward trends in empirical papers using other data sets and purely theoretical papers.6 The sharp shift toward experimental economic research is no doubt heavily due to the fact that Smith’s influential work legitimized the field.

  There is still the nagging question, though, about the extent to which economic experiments, conducted in laboratory settings, can be extrapolated into the real world with people’s own money on the line. Smith and his colleagues have shown that despite the skepticism, many various theoretical economic propositions can be confirmed through such experiments, while the experimental methods they have devised can be used more frequently in business settings.

  As I discuss shortly, businesses increasingly have been taking up that challenge, but in a different way than Smith and other experimentalists have gone about their work; that is, by using methods approximating the randomized controlled experiments in the field that are the gold standard for assessing the efficacy and dangers of pharmaceuticals or evaluating various social policy interventions. And businesses appear to have done so without being prodded by economists.

  Lab Experiments in Business: Focus Groups

  Before I get to that story, however, it is interesting to summarize how some businesses have used a similar laboratory approach to that used by the economic experimentalists. The experimental method to which I refer is the focus group. Movie studios use them to test potential audience reactions to different plots, especially endings. Advertising firms use this technique, along with politicians, to refine their messages, increasingly tailored to very different audiences. Some firms use focus groups to identify products or product areas that consumers might be attracted to. Go to the web and type in “focus groups in business” and among the entries that immediately pop up are articles suggesting that entrepreneurs use focus groups, formal or informal, to test the viability of their ideas before committing much time and a lot of money to new enterprises.

  There is a limit, of course, to the effectiveness of focus groups. By construction, they tend to be small, and thus may not be representative of larger populations. Also, what may seem to appeal in a small-group setting may not be readily accepted or welcomed when introduced into the wider marketplace, which is one reason companies may roll out new products or services in certain test geographic areas, getting feedback and introducing refinements before marketing to a much wider audience or market.

  At the same time, if they are properly run, focus groups can yield insights in ways that mass experiments, which often call for binary (yes or no) responses, cannot. Focus group participants may volunteer in an unstructured setting ideas or reactions unanticipated by the focus group organizers. This can lead to “aha” moments that change messaging or product designs, or even foster new ideas that firms were not originally planning to pursue.

  Economic Experiments in the Field: Randomized Controlled Trials

  Economists for years have borrowed techniques from the social sciences, engineering, and mathematics to model or infer individual or collective behaviors. In recent years, a small but growing band of economists, led initially by Michael Kremer of Harvard, and in a much more expansive fashion by Esther Duflo of MIT and Abhibit Banerjee of Harvard, has looked to a favorite technique used in medicine and program evaluation: randomized controlled experiments. Duflo, who earned her PhD and has since taught at MIT, won the Clark Medal in 2010 and a MacArthur genius award in 2009.

  RCTs compare treatment and control groups to test whether the tested medicine or policy intervention has had a statistically significant impact.7 Among the interventions the above-mentioned economists have tested are whether putting cameras in school rooms reduces teacher absences (it does), and whether microcredit programs encourage entrepreneurship in poor countries (also a yes).

  Kremer, Duflo, and Banerjee are not the only economists who conduct RCTs. Roland Fryer of Harvard uses the method to test the impact of various school interventions, such as paying kids in some manner for their achievements. The late Elinor Ostrom, a political scientist who won the Nobel Prize in Economics for her work (joining several other non-economists), conducted pioneering experiments validating that groups can self-organize to share resources.8 Tyler Cowen even argues in his new, important book Average Is Over that the future of economics, and indeed possibly all social sciences, lies in a merger into a single social science, in which researchers will specialize in the crunching of large bodies of data, some from experiments. The empirical results will drive the theory, not the other way around.9 I have more to say about this projection in Chapter 16.10

  One critique of RCT in economics and the social sciences relates to the transferability of the results from the specific experiments being conducted to other contexts. For example, it might be true that installing cameras in schoolrooms in India reduces teacher absences, but that result may be unique to a particular site or to wider region within India if not the entire country. It is not clear whether the same result would be obtained in different countries with different cultural norms and expectations. The same critique can be applied to other RCTs.

  Nonetheless, RCTs remain the gold standard not only in medicine but also in the evaluations of programs funded by foundations, schools, hospitals, and other organizations. Even if the results from one experiment in one setting are not generalizable to other organizations in other settings, the findings are likely to be used, and properly so, in the specific contexts where the tests are conducted.

  Business Experimentation in the Field

  Well before academic economists had discovered the use and power of RCTs, business began using real-world experiments, modeled loosely or closely on RCTs, to fine-tune their products, their marketing strategies, designs of their web pages, and so on. The pioneers on the front lines typically have not been economists, but they have acted like economists in using experimental methods to hone their business strategies and, in some cases, build great companies.

  James Manzi, one of the leaders in this field whose efforts I will describe shortly and who is profiled in the accompanying box, has outlined the history of business experimentation.11 In his telling, the true pioneers
of the practice were the founders of Capital One, today one of the nation’s largest credit card companies, whose success Manzi attributes to continuous and extensive experimentation.

  Capital One was launched by two former Strategic Planning Associates (SPA) consultants, Rich Fairbank and Nigel Morris, who attacked the credit card customer base as if it were one giant laboratory. Do customers respond better to solicitations in blue or white envelopes? Send out solicitations to two groups and see if there is a statistically significant difference. Apply the same approach to virtually everything else the company does, including employee selection, collection policies, cross selling of different financial products, and do as many as 60,000 experiments a year, and that is how Capital One was built. No single experiment was a “home run” but the “singles” from thousands of successful experiments incrementally improved the company so that in 25 years, Capital One grew from nothing into a major Fortune 500 company and one of the largest credit card lenders in the United States.

  That is Manzi’s personal experience with how business experiments work: They provide incremental rather than disruptive improvements that over time add up to either major breakthroughs or significant additions to companies’ bottom lines.

  The Internet unleashed a huge jump in business experimentation because it became so easy to do. Rather than sending out large batches of different colored envelopes and waiting for sales results, one can change the color or design of websites on a daily or even more frequent basis, and offer the different impressions to different sets of randomly chosen website visitors. If the comparisons are limited to just two sets of customers they now have a name—an A/B test—and we have Google to thank for introducing them.