by Andrew Leigh
6Jeremy Johnson, ‘Campaign experiment found to be in violation of Montana law’, Washington Post, 13 May 2015.
7Gueron & Rolston, Fighting for Reliable Evidence, pp. 48–9. They also point out the solution: ‘We made every effort to show local actors that what we were proposing was ethical and legal . . . complaints diminished and eventually disappeared, and the procedure became only a minor operational annoyance’.
8One analysis found that lack of implementation by teachers was a major reason why nine tenths of randomised trials commissioned by the US Department of Education reported no positive impact: Coalition for Evidence-Based Policy, ‘Randomized controlled trials commissioned by the Institute of Education Sciences since 2002: How many found positive versus weak or no effects?’, Washington, DC: Coalition for Evidence-Based Policy, 2013, cited in Abhijit Banerjee, Rukmini Banerji, James Berry, et al, ‘From proof of concept to scalable policies: Challenges and solutions, with an application’, Journal of Economic Perspectives, vol. 31, no. 4, 2017, pp. 73–102.
9Michael Hiscox, personal correspondence.
10Joan McCord, ‘The Cambridge-Somerville Study: A pioneering longitudinal-experimental study of delinquency prevention’ in Preventing Antisocial Behavior, edited by Joan McCord and Richard Tremblay, New York: Guilford, 1992, pp. 196–206.
11Stratified random sampling achieves a similar result, except that the sample is balanced in blocks rather than pairs.
12Dean Karlan & Jacob Appel, Failing in the Field: What We Can Learn When Field Research Goes Wrong, Princeton: Princeton University Press, 2016, p. 131.
INDEX
21st Century Community Learning Centers 73–4
see also Mathematica
‘95 per cent significance’ rule 195–8
‘1001 critical days’ movement 70
1964 Declaration of Helsinki 186
Aaron, Henry 61
‘A/B testing’ 130, 145, 155, 166, 167
Abdella, Teyiba 107–8
Abdul Latif Jameel Poverty Action Lab (J-PAL) 121–3
Abecedarian Project 68
ABRACADABRA reading program 189
AC Nielsen 140
Acayucan road experiment 110, 207
Accenture 139
Afghanistan village schools 118–19, 183, 187
agricultural field experiments 53, 126–7
Alhassan, Lariat 102–3
see also YouWiN!
Alm, Steven 97–8
Amazon 7, 131–2, 137, 144, 167
American Economic Association’s Randomized Controlled Trials registry 199
American Psychological Association, and Joseph Jastrow 51
Ames, Phil 184–5
‘anchoring’ effect 133
Andrade, Leandro 101
Annie Hall 208–9
Anson, Commodore George, and scurvy 1–3
Archilochus 104
Aristotle 156
artefactual field experiments 176
Arthroscopy 20
Ashenfelter, Orley 194
ASSISTments, and online learning tools 77
Auden, W.H. 125
Australian National University 167, 186, 201
‘automaticity’ 87–8
Badabla, Saa 161
Banerjee, Abhijit 121
Banerji, Rukmini 189–90
Barksdale, Jim 6
Battered Women’s Movement 89
Battlers and Billionaires 167
Belmont Report 186
Benin political campaign, and Saka Lafia 160
BETA 171
see also Michael Hiscox
Beyond Scared Straight 8
Bill & Melinda Gates Foundation 77, 103
biometrically identified smartcards 111
see also Karthik Muralidhan
Blane, Gilbert 4
Blattman, Chris 88, 209
Bloom, Nicholas 139
Bloomberg Philanthropies 211
Booker, Senator Cory 86
Borges, Jorge Luis, and ‘The Lottery in Babylon’ 181
Boston Consulting 139
Bown, Stephen 1
Brindley, Giles 168
Bristol, Muriel 52
British Medical Journal 12
‘broken windows policing’ 209
Broockman, David 163–4
Brookings Institution 44
see also Ron Haskins
Bush, President George W. 125, 155
California Votes Initiative 153
Campbell, Donald 205
Campbell Collaboration 8, 198
Capitol One 128–9
see also Nigel Morris; Rich Fairbank
Carlyle, Thomas 44
Carr, NSW Premier Bob 95
Chalmers, Iain 28
Chatterbooks program 76
see also Education Endowment Foundation (EEF)
‘cherrypick’ 36,196
Chicago ‘Parent Academy’ 9
childhood learning programs 65, 68, 70–2, 191
see also 21st Century Community Learning Centers; ‘1001 critical days’ movement; Abecedarian Project; Early Years Education Program; Head Start; Perry Preschool; Sesame Street; West Heidelberg early years centre
Choong, Peter 17–18, 21
Christoforetti, John 22
Cifu, Adam 31
Civis Analytics 159
Clark medal 121
see also Esther Duflo
Clinton, President Bill 41, 59, 105
Coby, Gary, and the Trump campaign 154
Cochrane 28, 31, 198
see also Archie Cochrane; Iain Chalmers
Cochrane, Archie 27–8, 190
Cochrane Collaboration 28
cognitive behavioural therapy 87–8
Colbert, Stephen 125
college program trials 82–3, 169
Commonwealth Scientific and Industrial Research Organisation (CSIRO) 14, 54
Community Led Total Sanitation 116
control groups 44, 78, 93, 118, 138, 192, 211
and ‘encouragement design’ 187
see also medical randomised trials; political campaign strategies; randomised trials; social field experiments
conventional laboratory experiments 176–7
see also Steven Pinker
Cook, James 4
Cook, Scott, and Intuit 207
Cooney, Joan 63
correlation 11, 106, 149
counterfactual 9–10, 74, 82, 149, 192–3
Cowdery, Nicholas 96
credible comparison groups 10, 37, 72
Creedence Clearwater Revival, and ‘Fortunate Son’ 42
CrimeSolutions.gov 101
criminal justice experiments
‘Becoming a Man’ program 87
and CrimeSolutions.gov 101
Hawaii’s Opportunity Probation with Enforcement (HOPE) 97–9
‘incapacitation effect’ 99
and Liberian experiment 88
Minneapolis Domestic Violence Experiment 90–1
Neighbourhood Watch 94, 183
and Street Narcotics Unit 92–3
and ‘three strikes’ law’ 99, 101
see also Drug Courts; policing programs; restorative justice experiments; US Police Foundation
CVS pharmacy 133–4
Danish State Serum Institute 26
‘data and safety monitoring boards’ 187
Deaton, Angus 6, 12, 124
‘deep canvassing’ 163–4
‘Development Innovation Ventures’ 210
see also Maura O’Neill; Michael Kremer
dibao 108
Dive, Roger 96
‘double-blind’ studies 26
driving licence experiment India 109
see also Sendhil Mullainathan
Drug Courts 95–6, 98, 182
Duflo, Esther 121–2, 206
Dukakis, Michael 151
Durkheim, Emile 91
Dziak, John 131
Early Years Education Program 71–2
Earned Income
Tax Credit 41
Easterly, William 112
eBay 130, 132
Edna McConnell Clark Foundation 211
Education Endowment Foundation (EEF) 75–6, 211
Einstein, Albert 208
Eliot, Charles 51
‘encouragement design’ 187
ethics 72, 98, 109, 118, 182, 186–8
see also medical randomised trials
ethics committees 186–7
see also 1964 Declaration of Helsinki; Belmont Report; ‘data and safety monitoring boards’
evidence-based medicine 24, 26–7
see also Alvan Feinstein; Archie Cochrane; David Sackett
‘experimental ideal’ 194
Experiments in Governance and Politics Network 199
Facebook 138, 143–5, 154–5
see also Sheryl Sandburg
Fairbank, Rich 128–9
family violence 85, 89–90
see also The Battered Women’s Movement; Minneapolis Domestic Violence Experiment
Farrer, William 127
Feinstein, Alvan 27
Fenner, Frank 168
Feynman, Richard, on scientific integrity 205
Fibiger, Johannes 26
Finckenauer, James 8
Finland government experiment 46
Fisher, Ronald 52–4, 127 see also ‘modern synthesis’
and ‘The evolution of sexual preference’ 53
‘fixed mindset’ 6
Fleischer, David 163–5
FlyBuys loyalty card, and Coles supermarket empire 134–5
Flynn, James, and the ‘Flynn effect’ 73
Food and Drug Administration 29, 188
Fox, President Vincente 117
framed field experiments 176
Frankfurt, Harry 205
Freedom from Hunger 212
French government experiment 46
Fryer, Roland 79
‘fuel league table’ 136
fundraising strategies 156–9
and the ‘lead donor’ 157
and the ‘once and done’ campaign 158
Salvation Army’s ‘Red Kettle Christmas drive 157
and the ‘Science of Philanthropy Initiative’ 159
Garner, Alan 183–4
see also Head Injury Retrieval Trial
Gawande, Atul, on ‘pointless medical care’ 34
‘general equilibrium’ effect 191
Gerber, Alan 149,151–3
German government unemployment incentive 45
Get Out the Vote 149
see also Alan Gerber; Donald Green
GiveDirectly 107
Glover, Danny 7
Gneezy, Uri 178
Goldacre, Ben 185
Google 7, 131, 141–4, 154, 208
and researching a book title 166–7
see also Eric Schmidt; Marissa Mayer
Gopnik, Alison 65–6
Gore, Al 155
Gosnell, Harold, and political campaigns 148–50
Graber, Ben 60
Grameen Bank 105
see also Muhammad Yunus
Green, Donald 149, 151–3, 163
see also Get Out the Vote
Grigg, Sue 37
see also ‘Journey to Social Inclusion’
Gueron, Judith 57–62, 206
see also Manpower Demonstration Research Corporation (MDRC); Project Independence
H&R Block 82
Halpern, David 171, 206
see also Nudge Unit
Harford, Tim 190, 205
Harlem Children’s Zone 79
Harrah’s casino 127–8
see also Gary Loveman
Harris, Ian, and medical randomised trials 21, 34
Harrison, Glenn, and categories of randomised experiments 176
Haskins, Ron 44
Hawaii’s Opportunity Probation with Enforcement (HOPE) 97–9
see also Steven Alm
‘Hawthorn effect’ 138
Haygarth, John, and the placebo effect 23–4
Head Injury Retrieval Trial 183–4
Head Start 192
‘healthy cohort’ effect 12
Highest Paid Person’s Opinion (HiPPO) 6
Hill, Austin Bradford 54–7
and Principles of Medical Statistics 55
and streptomycin trial 56
Hiscox, Michael 171
see also BETA
HMS Salisbury 3
HMS Wager 2
Holmes, Oliver Wendell Sr, and materia medica 25–6
‘incapacitation effect’ 99
Incredible Years Basic Parenting Programme 69
Innovations for Poverty Action 123
International Clinical Trials Registry Platform 199
Ioannidis, John 196–7
James Cook University 33
James Lind Alliance 28
see also Iain Chalmers
Jastrow, Joseph 50–1
J.B. Lawes’ Patent Manures 126
job seekers programs 170, 194
and the Finland government trial 46
and the French government trial 46
job training programs 9, 37, 44–5
see also Manpower Demonstration Research Corporation (MDRC); unemployment
John Templeton Foundation 159
Johnson, David, on single-centre trials 197
Johnson, Guy 36–7
see also ‘Journey to Social Inclusion’
Johnson, President Lyndon 41
‘Journey to Social Inclusion’ 37–8
see also Sacred Heart Mission
Kalla, Joshua 163–4
Kane, Evan 168
Karlan, Dean 103, 106, 146–7
Kelsey, Frances 188
and the ‘Kelsey Award’ 189
Kenyan electricity trial 110
Kenyan mini-bus driver experiments 115–16
Keynes, John Maynard 88, 203
Kleiman, Mark 99
kleroterion, and the lottery system 181
Kling, Jeffrey 209
Knight, Frank 175
Kohl’s 129
Kremer, Michael 206, 210
Lafia, Saka, and the Benin political campaign 160
Lammers, Rudi 84
and ‘restorative justice conferencing’ 85
Laura and John Arnold Foundation 211
Lawes, John Bennett, and J.B. Lawes’ Patent Manures 126–7
Lea, Tess, and ABRACADABRA 189–90
‘lead donor’ 157
LENA (‘language environment analysis’) 80
see also Thirty Million Words initiative
Levitt, Steven 181–2
Liberian experiment 88
Lind, James 3–6, 16, 23
and A Treatise of the Scurvy 3
Linos, Elizabeth, on ethics 186
List, John 159, 176
Loaded magazine 180
lottery system 11, 72, 172, 175
and Australian taxi licences 185
and the Dutch medical college 82
and the Hajj pilgrimage 174
and kleroterion 181
and Promise Academy 78–9
and the Vietnam war draft 42–3
and visas 173
Loveman, Gary 127–8
Ludwig, Jens 209
MacArthur Foundation genius grant 121
see also Esther Duflo
malaria prevention trials 111–13
see also William Easterly; World Health Organization
Manpower Demonstration Research Corporation (MDRC) 57–61
see also Judith Gueron; Project Independence
Massachusetts Institute of Technology 121
see also Abdul Latif Jameel Poverty Action Lab (J-PAL)
Massachusetts Medical Society 25
matatu 115–16
see also Kenyan mini-bus driver experiments
Mathematica 74–5
Mayer, Marissa 142
McArdle, Megan 207
McArthur, Alan, and bushfire research 15–16
McCain, John 146
McKenzie, David 103
see also YouWiN!
McKinsey 139
medical drug trials 26–31, 199–201
and clinical trials 28
see also ‘placebo effect’; streptomycin trial
medical randomised trials 3–6, 24–34, 183–4, 190, 201–2
and Archie Cochrane 27–8, 190
and control groups 13, 24
‘double-blind’ studies 26
and ethics 13, 21, 31, 56, 182, 184
and evidence-based medicine 26–7
and exercise science 33, 201–2
and Head Injury Retrieval Trial 183–4
and Iain Chalmers 28
and James Lind 3–6, 16, 23
and scurvy treatment trials 3–5, 16
‘single subject’ trials 168
and single-centre trials 197
and vaccinations 113
see also scurvy treatment trials
meta-analysis 85
Miami Herald 60
microcredit 5, 6, 105–6, 121, 123
microtargeting 141
Minchin, Tim, and ‘Storm’ 32–3
Minneapolis Domestic Violence Experiment 90–1
see also Lawrence Sherman
‘modern synthesis’ 53
see also Ronald Fisher
Moore, Evelyn 67
Morris, Nigel 128–9
motivation 107–8
‘Moving to Opportunity’ program 39–40
Moynihan, Senator Daniel Patrick 206
Mullainathan, Sendhil 109, 209
Muralidharan, Karthik 110–11, 123–4
Murphy, Patrick 89
see also US Police Foundation
Mycoskie, Blake 113
National Association for Music Education 76
National Council for Evaluation of Social Development 210
natural field experiments 176
neighbourhood project 38–40
see also Maricela Quintanar
Neighbourhood Watch 94, 98, 183
Netflix 7, 143
Netscape 6
see also Jim Barksdale
Newton, Isaac 203
‘Nimechill’ 120 see also sex education
No Child Left Behind Act 210
Nobel Peace Prize 105
see also Muhammad Yunus
N-of-1 168–9
Nosek, Brian, and replication of psychology studies 196
Nudge Unit 170–1, 206
see also David Halpern
Obama, President Barack 146–8, 159, 169
observational analysis 11
OkCupid, and the dating website trial 129–30
Okonjo-Iweala, Ngozi 102
ee also YouWiN!
Olds, David 211
‘once and done’ campaign, and Smile Train aid charity 158
O’Neill, John, and Black Saturday 2009 13–14
O’Neill, Maura 210
Oportunidades Mexico 117
see also President Vincent Fox
Oregon research on health insurance 42
parachute study, and randomised evaluation of 12