by Andrew Leigh
54According to the Center for Responsive Politics, the 2014 US election cycle saw 395 House incumbents running for re-election spend a total of $565 billion, while twenty-eight Senate incumbents running for re-election spent a total of $302 billion. For the presidential campaign, Barack Obama spent nearly $1 billion in 2008, and somewhat less in 2012. See www.opensecrets.org for details.
55Charles Lewis & Center for Public Integrity, The Buying of the Congress, New York: Avon Books, 1998, quoted in Joshua L. Kalla & David E. Broockman, ‘Campaign contributions facilitate access to congressional officials: A randomized field experiment’, American Journal of Political Science vol. 60, no. 3, 2016. pp. 545–58.
56Kalla & Broockman, ‘Campaign contributions’.
57Daniel M. Butler & David W. Nickerson, ‘Can learning constituency opinion affect how legislators vote? Results from a field experiment’, Quarterly Journal of Political Science, vol. 6, 2011, pp. 55–83.
58Daniel E. Bergan, ‘Does grassroots lobbying work? A field experiment measuring the effects of an e-mail lobbying campaign on legislative behavior’, American Politics Research, vol. 37, 2009, pp. 327–52.
59Brendan Nyhan & Jason Reifler, ‘The effect of fact checking on elites: A field experiment on US state legislators’, American Journal of Political Science, vol. 59, no. 3, 2015, pp. 628–40. Another modest intervention that induced significant behavioural change was a study that worked with Republican Party leaders to send letters to neighbourhood precinct chairs, encouraging them to include more women in their delegation. The most powerful intervention increased the share of women from 24 to 30 per cent: Christopher F. Karpowitz, J. Quin Monson & Jessica Robinson Preece, ‘How to elect more women: Gender and candidate success in a field experiment’, American Journal of Political Science, vol. 61, no. 4, 2017, pp. 927–43.
60Quoted in Angus Chen, ‘Study finds deep conversations can Reduce transgender prejudice’, Health Shots, NPR Radio, 7 April 2016.
61Michael J. LaCour & Donald P. Green, ‘When contact changes minds: An experiment on transmission of support for gay equality’, Science, vol. 346, no. 6215, 2014, pp. 1366–9.
62David Broockman, Joshua Kalla & Peter Aronow, ‘Irregularities in LaCour (2014)’, Working Paper, 2015.
63Quoted in Maria Konnikova, ‘How a gay-marriage study went wrong’, New Yorker, 22 May 2015.
64Quoted in Chen, ‘Study finds’.
65Quoted in Chen, ‘Study finds’.
66David Broockman & Joshua Kalla, ‘Durably reducing transphobia: A field experiment on door-to-door canvassing’, Science, vol. 352, no. 6282, 2016, pp. 220–4. See also Ian Chipman, ‘Fighting transphobia in 10 minutes’, Stanford Business Insights, 7 April 2016.
67Quoted in Kathleen Maclay, ‘UC Berkeley, Stanford study finds canvassing conversations reduce transgender prejudice’, Berkeley News, 7 April 2016.
10 TREAT YOURSELF
1This is the price range from third party sellers, as listed at CamelCamelCamel.com. The example of Classic Twister is drawn from Jerry Useem, ‘How online shopping makes suckers of us all’, The Atlantic, May 2017.
2See Lawrence K. Altman, Who Goes first?: The Story of Self-Experimentation in Medicine, New York: Random House, 1987.
3Laurence Klotz, ‘How (not) to communicate new scientific information: A memoir of the famous Brindley lecture’, BJU international, vol. 96, no. 7, 2005, pp. 956–7.
4See, for example, Paul A. Scuffham, Jane Nikles, Geoffrey K. Mitchell, et al., ‘Using n-of-1 trials to improve patient management and save costs’, Journal of General Internal Medicine, vol. 25, no. 9, 2010, pp. 906–13.
5On the trial, see Stephanie S. Weinreich, Charlotte Vrinten, Jan J.G.M. Verschuuren, et al., ‘From rationing to rationality: An n-of-one trial service for off-label medicines for rare (neuromuscular) diseases’, Orphanet Journal of Rare Diseases, vol. 7, no. 2, 2012, p. A29. On prevalence and incidence of rare neuromuscular diseases, see Michael Rubin, ‘How common are neuromuscular disorders?’ Neurology Alert, vol. 34, no. 7, 2015, pp. 53–4.
6Megan Brooks, ‘Rare disease treatments make up top 10 most costly drugs’, Medscape, 2 May 2017.
7Coalition for Evidence-Based Policy, ‘Memorandum: Announcing Winners of the Coalition’s Low-Cost RCT Competition’, 15 July 2014.
8The announcement was made in 2015. In the same year, the Coalition for Evidence-Based Policy was subsumed into the Laura and John Arnold Foundation. The funding range for the low-cost randomised experiment competition was also amended to $100,000–$300,000. See Laura and John Arnold Foundation, ‘Laura and John Arnold Foundation announces expanded funding for low-cost randomized controlled trials to drive effective social spending’, press release, 7 December 2015.
9David Halpern, Inside the Nudge Unit: How Small Changes Can Make a Big Difference, London: WH Allen, 2015, p. 274.
10Halpern, Inside the Nudge Unit, p. 274.
11Halpern, Inside the Nudge Unit, pp. 91–2.
12Halpern, Inside the Nudge Unit, p. 89.
13Halpern, Inside the Nudge Unit, pp. 113–15; Michael Hallsworth, John List, Robert Metcalfe & Ivo Vlaev, ‘The behavioralist as tax collector: Using natural field experiments to enhance tax compliance’, Journal of Public Economics, vol. 148, issue C, 2017, pp. 14–31. The Australian Taxation Office reports that it increased compliance from tax debtors by 5 per cent simply by removing from its letters the opening words: ‘Please disregard this letter if you have paid this debt in full in the last seven days’: Peter Martin, ‘Mind games could pay handsomely’, Sydney Morning Herald, 17 November 2013.
14Halpern, Inside the Nudge Unit, p. 90.
15Tim Harford, ‘Nudge, nudge. Think, think. Say no more …’, Financial Times, 11 February 2012
16Halpern, Inside the Nudge Unit, p. 132.
17A reminder message that included the name of the client, the name of the adviser, and the words ‘Good luck!’ increased attendance rates from 10 per cent to 27 per cent: David Halpern, Inside the Nudge Unit: How Small Changes Can Make a Big Difference, WH Allen, London, 2015, pp. 120–2
18Halpern, Inside the Nudge Unit, pp. 275–8.
19Halpern, Inside the Nudge Unit, p. 340. In France, the Youth Ministry has set up an experimental laboratory, ‘Le Fonds d’Expérimentation pour la Jeunesse’, to test programs that help young people: see http://experimentation.jeunes.gouv.fr.
20Premier and Cabinet Behavioural Insights Team, ‘Understanding people, better outcomes: Behavioural insights in NSW’, Sydney: NSW Department of Premier and Cabinet, 2014.
21The control message was: ‘You have an appointment with [Doctor name] in [Clinic name] on [Date] at [Time]. For enquiries, call 8382-3150. Do not reply.’ The most effective treatment added the words: ‘If you attend the hospital will not lose the $125 we lose when a patient does not turn up.’ See Paul Herbert, Joyce Nathaney, Simon Raadsma & Alex Gyani, ‘Reducing missed outpatient appointments at St Vincent’s Hospital Sydney’, Sydney: St Vincent’s Hospital Sydney and NSW Department of Premier and Cabinet, 2015.
22Interview with Michael Hiscox, 4 August 2016.
23Peter Kuhn, Peter Kooreman, Adriaan Soetevent & Arie Kapteyn, ‘The effects of lottery prizes on winners and their neighbors: Evidence from the Dutch postcode lottery’, American Economic Review, vol. 101, no. 5, 2011, pp. 2226–47.
24George Bulman, Robert Fairlie, Sarena Goodman & Adam Isen, ‘Parental resources and college attendance: Evidence from Lottery Wins’, NBER Working Paper 22679, Cambridge, MA: National Bureau of Economic Research, 2016.
25David Cesarini, Erik Lindqvist, Matthew J. Notowidigdo & Robert Östling, ‘The effect of wealth on individual and household labor supply: Evidence from Swedish Lotteries’, NBER Working Paper 21762, Cambridge, MA: National Bureau of Economic Research, 2015.
26Gallup World Poll survey, quoted in Paul Collier, Exodus: How Migration is Changing Our World, Oxford: Oxford University Press, 2013, p. 167.
27For a discussion of this literature, see David McKenzie, ‘Lear
ning about migration through experiments’ in Christian Dustmann (ed.), Migration: Economic Change, Social Challenge, Oxford: Oxford University Press, 2015.
28Michael A Clemens, ‘Why do programmers earn more in Houston than Hyderabad? Evidence from randomized processing of US visas’, American Economic Review, vol. 103, no. 3, 2013, pp. 198–202.
29David McKenzie, Steven Stillman & John Gibson, ‘How important is selection? Experimental vs. non-experimental measures of the income gains from migration’, Journal of the European Economic Association, vol 8, no. 4, 2010, pp. 913–45.
30David Clingingsmith, Asim Ijaz Khwaja & Michael R. Kremer, ‘Estimating the impact of the Hajj: Religion and tolerance in Islam’s global gathering’, Quarterly Journal of Economics, vol. 124, no. 3, 2009, pp. 1133–70.
31Until 1995, the US program was known as the Taxpayer Compliance Measurement Program; it is now the National Research Program. For a valuable history, see Wendy Rotz, J. Murlow & Eric Falk, ‘The 1995 Taxpayer Compliance Measurement Program (TCMP) sample redesign: A case history’, Turning Administrative System Into Information System. Internal Revenue Service, Washington, 1994, pp. 699–703; Andrew Johns & Joel Slemrod, ‘The distribution of income tax noncompliance’, National Tax Journal, vol. 63, no. 3, 2010, pp. 397–418.
32OECD Forum on Tax Administration – Compliance Sub-Group, ‘Compliance Risk Management: Use of Random Audit Programmes’, Paris: OECD, 2004. Australia has recently embarked upon a small-scale randomised audit program: Nassim Khadem, ‘Tax man to hit SMEs and individuals with random audits’, Sydney Morning Herald, 5 November 2015. However, the Australian Inspector-General of Taxation has recommended a more ambitious approach: see Inspector-General of Taxation, Review into Aspects of the Australian Taxation Office’s Use of Compliance Risk Assessment Tools: A Report to the Assistant Treasurer, Canberra: Australian Government, 2013, pp. 145–7.
33OECD, ‘Compliance Risk Management’.
34Andrew Johns & Joel Slemrod, ‘The distribution of income tax noncompliance’, National Tax Journal, vol. 63, no. 3, 2010, pp. 397–418. The average underreporting of adjusted gross income is 3.8 per cent for the bottom 50 per cent of taxpayers, and 17 per cent for the top 1 per cent of taxpayers. By contrast to underreporting of income, underreporting of tax is higher among lower-income taxpayers. Note too that audit studies can miss income in tax havens, which is strongly skewed towards the top: Annette Alstadsæter, Niels Johannesen & Gabriel Zucman, ‘Tax Evasion and Inequality’, NBER Working Paper No. 23772, Cambridge, MA: NBER, 2017.
35Eric Avis, Claudio Ferraz & Frederico Finan, ‘Do government audits reduce corruption? Estimating the impacts of exposing corrupt politicians’, Journal of Political Economy, forthcoming.
36F.H. Knight, Risk, Uncertainty, and Profit, New York: Cosimo, 1921, p. 313, quoted in Omar Al-Ubaydli & John A. List, ‘On the generalizability of experimental results in economics’, in Guillaume R. Fréchette and Andrew Schotter (eds) Handbook of Experimental Economic Methodology, New York: Oxford University Press, 2015, pp. 420–62.
37Glenn W.Harrison & John A. List, ‘Field experiments’, Journal of Economic Literature, vol. 42, no. 4, 2004, pp. 1009–55.
38John A. List, ‘Do explicit warnings eliminate the hypothetical bias in elicitation procedures? Evidence from field auctions for sportscards”, American Economic Review, vol. 91, no. 4, 2001, pp. 1498–1507.
39Peter Bohm, ‘Estimating the demand for public goods: An experiment”, European Economic Review, vol. 3, 1972, pp. 111–30.
40Quoted in Manzi, Uncontrolled, p. 152.
41Robert Slonim, Carmen Wang, Ellen Garbarino & DanielleMerrett, ‘Opting-In: Participation Biases in Economic Experiments’, Journal of Economic Behavior and Organization, vol. 90, 2013, pp. 43–70.
42Ernst Fehr & John A. List, ‘The hidden costs and returns of incentives – trust and trustworthiness among CEOs”, Journal of the European Economic Association, vol. 2, no. 5, 2004, pp. 743–71.
43Steven D. Levitt & John A. List, ‘What do laboratory experiments measuring social preferences reveal about the real world?’ Journal of Economic Perspectives, vol. 21, no. 2, 2007, pp. 153–74.
44Arthur Aron, Edward Melinat, Elaine N. Aron, Robert Darrin Vallone & Renee J. Bator, ‘The experimental generation of interpersonal closeness: A procedure and some preliminary findings’, Personality and Social Psychology Bulletin, vol. 23, no. 4, 1997, pp. 363–77.
45Gneezy & List, The Why Axis, pp. 224–6.
11 BUILDING A BETTER FEEDBACK LOOP
1Luke Rhinehart is also known as George Cockcroft. The story is told in his interview with Andrew Denton, Enough Rope, ABC TV, 27 September 2004.
2Jorge Luis Borges, Collected Fictions, translated by Andrew Hurley, New York: Penguin Putnam, 1998, pp. 101–6.
3In The Luck of Politics, I discussed the many ways in which chance affects political careers. Andrew Leigh, The Luck of Politics, Melbourne: Black Inc, 2015.
4These results are from Steven Levitt, ‘Heads or tails: The impact of a coin toss on major life decisions and subsequent happiness’, NBER Working Paper No. 22487, Cambridge, MA, : NBER, 2016,.
5Stephen Dubner & Steven Levitt, Think Like a Freak, New York: William Morrow, 2014, p. 201.
6Dubner & Levitt, Think Like a Freak, p. 203.
7Andrew Leigh, ‘A good test of public policy’, Australian Financial Review, 8 April 2008, p. 70.
8Alan A. Garner, Kristy P. Mann, Michael Fearnside, Elwyn Poynter & Val Gebski, ‘The head injury retrieval trial (HIRT): A single-centre randomised controlled trial of physician prehospital management of severe blunt head injury compared with management by paramedics only’, Emergency Medicine Journal, vol. 32, no. 11, 2015, pp. 869–75. My description is based on the ‘intent to treat’ estimates (that is, comparing groups based on initial random assignment). I do not follow the authors in discussing ‘as treated’ estimates, since these are not necessarily based on random assignment and may therefore be biased.
9Alan A. Garner, Michael Fearnside & Val Gebski, ‘The study protocol for the Head Injury Retrieval Trial (HIRT): a single centre randomised controlled trial of physician prehospital management of severe blunt head injury compared with management by paramedics’, Scandinavian Journal of Trauma, Resuscitation and Emergency Medicine, vol. 21, no. 1, article 69, 2013.
10Asked whether they support ‘The use of controlled experiments or trials to design and test more areas of government social policy’, 73 per cent of Australian parliamentarians and 67 per cent of UK parliamentarians answer ‘Strongly support’ or ‘Tend to support’. Results are based on 104 British MPs interviewed in 2014 and 109 Australian MPs (territory, state and federal) surveyed in 2016. Australian results in Phil Ames & James Wilson, ‘Unleashing the potential’, PAE prepared for client Andrew Leigh, Cambridge, MA: Harvard Kennedy School, 2016. British results in Ipsos MORI, ‘Are MPs open to experimenting?’, London: Ipsos MORI, 2015.
11Forty-eight per cent of Australian politicians and 35 per cent of British politicians agreed with the statement ‘Randomly choosing whether some people get a policy intervention and others do not is unfair’. By contrast, just 10 per cent of Australian politicians and 9 per cent of British politicians agreed with the statement ‘Controlled experiments or trials are too expensive as a way of designing and testing social policies’. See Ames & Wilson, ‘Unleashing the potential’; Ipsos MORI, ‘Are MPs open to experimenting?’, London: Ipsos MORI, 2015. See also a small survey of Australian public servants which found that twenty-four out of twenty-seven believed that random assignment was ethical: Kyle Peyton, ‘Ethics and politics in field experiments’, The Experimental Political Scientist, vol 3, no. 1, 2012, pp. 20–37.
12Quoted in Ipsos MORI, ‘What do MPs think of randomised controlled trials (RCTs)?’, London: Ipsos MORI, 2015.
13See, for example, ‘Hundred more taxis in city soon’, Sydney Morning Herald, 19 July 1946, p. 3; ‘Ballot for new taxi licences’, Argus, 20 September 1946, p. 4.
14Rachel Glennerster, ‘The practicalities of running randomized evaluations: Partnerships, measurement, ethics, and transparency’ in Banerjee and Duflo (eds), Handbook of Field Experiments, pp. 175–243.
15Quoted in Ames & Wilson, ‘Unleashing the potential’.
16Glennerster, ‘The practicalities of running randomized evaluations’.
17Alfredo R. Paloyo, Sally Rogan & Peter Siminski, ‘The effect of supplemental instruction on academic performance: An encouragement design experiment’, Economics of Education Review, vol. 55, 2016, pp. 57–69.
18Quoted in Gardiner Harris, ‘The public’s quiet savior from harmful medicines’, New York Times, 13 September 2010, p. D1.
19Quoted in Glennerster, ‘The practicalities of running randomized evaluations’.
20Tess Lea, ‘Indigenous education and training: what are we here for?’ in Jon Altman & Melinda Hinkson (eds), Culture Crisis: Anthropology and Politics in Remote Aboriginal Australia, Sydney: UNSW Press, 2010, pp. 195–211. In addition, the research team faced an environment in which about half of all Northern Territory teachers quit in a given year, in which the education department had three chief executives in three years, and in which up to one in five students move school each year. Janet Helmer, Helen Harper, Tess Lea, et al., ‘Challenges of conducting systematic research in Australia’s Northern Territory’, Asia Pacific Journal of Education, vol. 34, no. 1, 2014, pp. 36–48.
21The results are available at Jennifer Wolgemuth, Janet Helmer, Helen Harper, et al., ABRACADABRA (ABRA) Early Childhood Literacy Project, Annual Report No. 3. A Multi-Site Randomised Controlled Trial and Case Study of the ABRA Literacy Software in NT Schools, Darwin: Menzies School of Social Research, 2011. As to her sense of resignation afterwards, Lea wrote: ‘In fact, after years of tilting at indigenous education, I am conceding defeat. I have worked out it requires more of me than I am prepared to give.’: Tess Lea, ‘Indigenous education and training: what are we here for?’, Presentation to the Department of Education, Employment and Workplace Relations, Canberra, 1 March 2010. For similar problems in the context of a planned randomised trial on strategies to curb alcohol misuse, see Beverly M. Sibthorpe, Ross S. Bailie, Maggie A. Brady, et al., ‘The demise of a planned randomised controlled trial in an urban Aboriginal medical service’, Medical Journal of Australia, vol. 176, no. 6, 2002, pp. 273–6.