by Andrew Leigh
PRAISE FOR RANDOMISTAS
‘Randomistas is a tour de force – an engaging, passionate, how-to account of randomised experiments. After reading Leigh’s book, you’ll be baffled at the many businesses and governments yet to catch on. Fortunately, Leigh also offers a simple guide that anyone can follow. If the next generation of policymakers follows his advice – and let’s hope they do – this book will literally change the world.’
DAVID HALPERN, head of the UK’s Behavioural Insights Team, author of Inside the Nudge Unit
‘Randomistas takes the reader on a great journey about how data and experiments can make the world a better place, one policy at a time.’
DEAN KARLAN, professor of economics at Yale
‘The subject of this book could hardly be more vital: are we humble enough to admit we may be wrong, and do we care enough to learn? Randomistas is rigorous, impassioned and tremendous fun. Everyone should read it.’
TIM HARFORD, author of The Undercover Economist and Fifty Things That Made The Modern Economy
‘Packed with tantalising tales, Randomistas is essential reading for anyone interested in debunking myths and uncovering hidden truths.’
STEVEN LEVITT, co-author of Freakonomics
ANDREW LEIGH is the Shadow Assistant Treasurer and Federal Member for Fenner in the Australian Parliament. Prior to being elected in 2010, he was a professor of economics at the Australian National University. Andrew holds a PhD in public policy from Harvard, having graduated from the University of Sydney with first-class honours in Law and Arts. He is a Fellow of the Australian Academy of Social Sciences and a past recipient of the Economic Society of Australia’s ‘Young Economist Award’. Andrew is a keen marathon runner and hosts a podcast titled ‘The Good Life’ on living a happy, healthy and ethical life. He lives in Canberra with his wife, Gweneth, and their three sons.
BY THE SAME AUTHOR:
Choosing Openness: Why Global Engagement is Best for Australia
The Luck of Politics
The Economics of Just About Everything
Battlers and Billionaires: The Story of Inequality in Australia
Disconnected
Published by La Trobe University Press in conjunction with Black Inc.
Level 1, 221 Drummond Street
Carlton VIC 3053, Australia
[email protected]
www.blackincbooks.com
www.latrobeuniversitypress.com.au
Copyright © Andrew Leigh 2018
Andrew Leigh asserts his right to be known as the author of this work.
ALL RIGHTS RESERVED.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form by any means electronic, mechanical, photocopying, recording or otherwise without the prior consent of the publishers.
9781863959711 (paperback)
9781743820278 (ebook)
Cover design by Design by Committee
Cover image: Bigstock
Text design and typesetting by Marilyn de Castro
CONTENTS
1.Scurvy, Scared Straight and Sliding Doors
2.From Bloodletting to Placebo Surgery
3.Decreasing Disadvantage, One Coin Toss at a Time
4.The Pioneers of Randomisation
5.Learning How to Teach
6.Controlling Crime
7.Valuable Experiments in Poor Countries
8.Farms, Firms and Facebook
9.Testing Theories in Politics and Philanthropy
10.Treat Yourself
11.Building a Better Feedback Loop
12.What’s the Next Chance?
Ten Commandments for Running Your Own Randomised Trial
Acknowledgements
Endnotes
Index
1
SCURVY, SCARED STRAIGHT AND SLIDING DOORS
As Commodore George Anson’s six warships rounded the tip of South America, he knew his plans were unravelling. Commissioned by the British government to capture Spanish colonies in Panama and Peru, Anson had been at sea for half a year. The seas were rough and a Spanish squadron was chasing him, but the greatest threat to the men on board was a disease: scurvy.
Anson’s expedition left England in 1740 with 1854 men. Four years later, just 188 of them returned. Of the eight ships that set out, only one made it back. Writer Stephen Bown describes the voyage as the worst medical disaster ever to take place at sea. Six months in, men were dying daily. At first, the corpses were sewn into their hammocks and thrown overboard. But after a time the living sailors became too weak to deal with the dead, and they were left below deck, where they had drawn their last breaths.
Sailing an eighteenth-century warship required many men. Anticipating that some would die of scurvy, captains often took along extra personnel. But Anson’s losses exceeded expectations. With few men having the strength to work, some of the warships crashed into rocks off South America. One ship, the Wager, was ripped apart on the jagged rocks of southern Chile. Some of the sailors drowned because they were too weak to swim ashore.
Scurvy affects the body’s connective tissues. At first, victims feel tired and uncoordinated. They bruise easily and their legs begin to swell. Then their gums become inflamed, their breath grows foul and their skin becomes blotchy. Sailors were shocked to see old battle wounds beginning to bleed and bones that had healed separating again. An anonymous surgeon wrote of his own afflictions with the disease: ‘It rotted all my gums, which gave out a black and putrid blood. My thighs and lower legs were black and gangrenous, and I was forced to use my knife each day to cut into my flesh in order to release this black and foul blood . . . And the unfortunate thing was that I could not eat, desiring more to swallow than to chew.’1 In the final stages, patients’ gums swell up so much they cannot eat. Internal bleeding leads to death.
During the age of sail, more than 2 million sailors died from scurvy, more than from skirmishes, storms and shipwrecks combined.2 In 1499 Portuguese explorer Vasco da Gama lost over half his crew to scurvy. In 1520 his countryman Ferdinand Magellan lost two-thirds of his crew to the disease.3 In the Seven Years’ War (1756–63), Britain raised 185,899 sailors; only 1512 died in action, while 133,708 died of scurvy.4 Put another way, a British sailor in the Seven Years’ War had less than a 1 per cent chance of being killed in action, but a 72 per cent chance of dying from scurvy.
Everyone had their own theory on how to fight the disease, with doctors variously proposing wine, ginger and salts. But because none of the remedies was carefully tested, scurvy continued to run rampant. In the absence of evidence, whether a doctor’s crackpot solution was adopted depended on his status and confidence. Sailors continued to die in their thousands.
Then, in 1747, a 31-year-old ship’s surgeon by the name of James Lind ran a remarkable experiment. Several months into the voyage of the HMS Salisbury, most of the crew were affected by scurvy. Lind decided to try different treatments on twelve sailors with advanced scurvy, ‘as similar as I could have them’. He took other people’s theories and found a way to test them. Lind’s randomised trial tested six treatments, with each being given to a pair of sailors. The first group got a litre of cider, the second 4 millilitres of sulphuric acid (‘elixir of vitriol’) and the third 80 millilitres of vinegar. If those last two treatments weren’t bad enough, the fourth group were made to drink 250 millilitres of seawater, while the fifth group got a mixture of nutmeg, garlic, mustard-seed, horseradish, balsam of Peru and gum myrrh. The sixth group received two oranges and one lemon. Apart from the treatments, all the patients were fed the same diet and kept in the same part of the ship.5
It didn’t take long for the experiment to show results. Lind reported that ‘the most sudden and visible good effects were perceiv
ed from the use of oranges and lemons’, to the extent that one of the patients who had been given citrus fruit was ready to return to duty in less than a week. By contrast, the patients given acid, vinegar and sea-water did not improve. Given that sulphuric acid was the British Navy’s main treatment for scurvy, this was an important finding.
The results were clear, but, as any researcher knows, a convincing result doesn’t immediately lead policymakers to change their minds. It took Lind six years to write up the results, in a book titled A Treatise of the Scurvy. The 456-page work was dedicated to Commodore George Anson, whose expedition had lost nine-tenths of its men. Unfortunately, although Lind’s experimental results were spot-on, his theoretical explanations for why citrus worked were hocus-pocus.6 The treatise was largely ignored in the years after its publication.
For the next few decades, scurvy remained the single greatest risk to long sea voyages. The longest sea voyage that the British government was contemplating was the one to Australia. Scurvy showed its symptoms within a couple of months, and the voyage to Australia took six to twelve months.
By luck, both the exploratory voyage of James Cook in 1768–71 and the First Fleet of Arthur Phillip in 1787–88 stumbled upon ways to keep scurvy at bay. Cook gave his men sauerkraut, malt and lemon juice, and called in to port every two or three months for fresh food. He wrongly thought the main preventatives were sauerkraut and malt, but nonetheless managed to return to London without losing a single sailor to scurvy.7 Like Cook, Phillip’s expedition mistakenly thought sauerkraut and malt could ward off scurvy.8 But the passengers on the First Fleet were saved from its worst ravages by frequent stops along the way. The convicts ate fresh food at Portsmouth before leaving Britain, and then en route at Tenerife, Rio de Janeiro and the Cape of Good Hope. Even so, the disease threatened the voyage. On 26 January 1788, after raising the English flag at Port Jackson, Phillip wrote in his diary that ‘the scurvy began to rage with a virulence which kept the hospital tents generally supplied with patients’.9
Cook and Phillip were lucky, but scurvy returned with a vengeance on the Second Fleet (1789–90), which had a death rate of nearly one in three.
Then, in the 1790s, a disciple of Lind, surgeon Gilbert Blane, was able to persuade senior naval officials that oranges and lemons could prevent scurvy.10 In 1795 – almost fifty years after Lind’s findings – lemon juice was issued on demand; by 1799 it became part of the standard provisions.11 In the early 1800s British naval sailors were consuming 200,000 litres of lemon juice annually.12
The British may have been slow to adopt Lind’s findings, but they were faster at curing scurvy than their main naval opponents. An end to scurvy was a key reason why the British, under the command of Admiral Lord Nelson, were able to maintain a sea blockade of France and ultimately win the 1805 Battle of Trafalgar against a larger force of scurvy-ridden French and Spanish ships. Unlike Commodore Anson six decades earlier, Lord Nelson didn’t have to fight while scurvy ravaged his crew.
In the age of sail, over 160,000 convicts and many more free settlers made their way to Australia. Had James Lind not found a way of preventing scurvy, many of those settlers – ancestors of present-day Australians – would have died at sea. In fact, it is possible that if the scurvy randomised trial had been conducted by another colonial superpower, Australia’s national language might be French, Dutch or Portuguese.
*
You might think some things are so obvious we don’t need randomised trials to prove them.
•If you have bad back pain, an X-ray will help pinpoint the problem.
•To discourage early pregnancy, ask teenage girls to care for a baby doll that’s programmed to demand attention at all hours.
•After-school programs are a great way to help troubled youths.
•Microcredit can solve world poverty.
•If you want people in developing countries to sleep under a bed net to prevent insect-borne disease, don’t just give it to them – ask them to pay for it, so that they value it.
Each of these statements sounds completely reasonable, doesn’t it? Unfortunately, all five claims are completely wrong. For non-specific back pain, X-rays don’t help. Girls who cared for an infant simulator for a week were twice as likely to become teenage mothers.13 Many after-school programs have no measurable impact. Rigorous microcredit studies have found it makes only a small impact. Free distribution of bed nets massively increases take-up.
Sometimes randomised trials do confirm conventional wisdom – but their real value is when they surprise us. Talk to people who run a lot of experiments, and you’ll discover a sense of scepticism about how far gut feel can take us. In a world as complicated as ours, one of the things I admire about those who run randomised trials is that they’re modest about their understanding of the world. Many embody the philosophy of Albert Einstein, who once said, ‘The more I learn, the more I realise how much I don’t know.’ The recognition that we can use our failures to do better next time has been dubbed a ‘growth mindset’.14 It contrasts with a ‘fixed mindset’, in which we fear setbacks because we regard our talents and abilities as static. People who have a growth mindset know that we can become smarter over time, so long as we take the effort to learn what works and what does not.
Like James Lind, these ‘randomistas’ – as Nobel laureate Angus Deaton once dubbed them15 – have an independent streak. As former Netscape head Jim Barksdale liked to joke to his staff, ‘If we have data, let’s look at data. If all we have are opinions, let’s go with mine.’ Randomistas know that the alternative to using rigorous evidence is often to follow the HiPPO – the Highest Paid Person’s Opinion. In Africa, the hippo is the most dangerous large animal on the continent; HiPPOs can be just as deadly. Randomised trials can save lives, whereas privileging hunches over facts can be lethal.
Today, randomised trials are being run for all kinds of unexpected purposes. Researchers in the Netherlands randomly assigned primary school students to a sports program run by one of the nation’s top soccer teams, to see if it helped them perform better in maths and reading (it didn’t).16 In Washington DC, researchers randomly offered a Washington Post subscription to households to see how it affected their political views (it made them more likely to vote Democrat).17 A French experiment found that winning a spot in a boarding school boosted test scores for disadvantaged students.18 A team of economists used a randomised trial in India to test whether better cooking stoves would improve health by reducing indoor air pollution (the effect was temporary, lasting only a year or so).19 In Ethiopia, a randomised trial tested whether getting a job in a sweatshop improved people’s lives (most quit within a few months).20 In Oregon, trials have compared whether delinquent youths do better in foster care or group care (foster care seems to be better, particularly for girls).21
Randomised trials are in your life, whether you like it or not. In most advanced countries, governments won’t pay for pharmaceuticals unless they’ve undergone a randomised evaluation. Increasingly, the world’s smartest aid agencies are looking for the same level of evidence before they allocate funds to a project. Did you use the internet today? Congratulations, you’ve most likely participated in several randomised trials. Netflix, Amazon and Google are constantly using experiments to hone their sites.
Randomised trials have the power to surprise us. In 1978 a documentary film set in a New Jersey jail hit American screens. Narrated by a young Danny Glover, Scared Straight brought a group of juvenile offenders face to face with hardened criminals. The youngsters were supposed to be ‘scared’ onto the straight and narrow by the tough-talking prisoners, who described life behind bars. In one scene, the young people were ordered to take their shoes off so they could feel what it was like to have their possessions taken away. One of the convicts then growled, ‘You know if you get up and touch one of them shoes, I’m gonna break my leg off in your ass.’
Scared Straight not only won an Oscar, it also inspired policymakers across America to se
t up Scared Straight programs. Mostly, these policymakers relied on anecdotes as their evidence. Sometimes they pointed to low-quality evaluations, which compared participants with youths who refused to take part. These studies concluded that Scared Straight cut crime by up to half.22
If policymakers had listened to more careful assessments, they might have been a little more sceptical. As early as 1978, criminologist James Finckenauer conducted the first randomised evaluation of the Scared Straight program.23 ‘The evidence showed that the kids who went on the program were at greater risk of offending than those who didn’t.’24 As researchers like to say, ‘data is not the plural of anecdote’.
Many ignored Finckenauer’s randomised study, but over time further rigorous evaluations were carried out and reached the same conclusion. In 2002 the non-profit Campbell Collaboration put out a systematic review of the evidence.25 It reported that Scared Straight didn’t cut crime; it increased it by as much as one-quarter. Furthermore, some of the young participants reported that inmates stole from them and sexually propositioned them.
Like the Terminator, Scared Straight has been hard to kill. In 2011, A&E aired a television program called Beyond Scared Straight, which continued to perpetuate the myth that Scared Straight works.
Humans love a good story. But statistics help us tell the difference between fact and fairytale. Scared Straight makes a cracking tale, but at its heart is a myth. However, since stories are often more appealing to us than dry evidence, it took decades for governments to finally scrap it.
*
Another example of randomised trials upending accepted wisdom is in job training for disadvantaged youth. In the mid-1980s the US government commissioned a major randomised trial of job training programs. The experiment found that young people who participated in the programs earned less over the subsequent three years than those who didn’t get the training.26 Other randomised trials found that while job training did no harm, it didn’t do much good either.27