Randomistas

by Andrew Leigh

  To help answer this question, the US government had set up a randomised experiment known as the ‘Moving to Opportunity’ program. In five cities – Baltimore, Boston, Chicago, New York and Los Angeles – thousands of people living in high-poverty neighbourhoods signed up to be part of a program that gave them a chance of moving to a better area. They were then randomly assigned to one of three groups. One group was given a subsidised housing voucher and required to move to a low-poverty neighbourhood. Another group was given a housing voucher with no strings attached. A control group did not receive any voucher, and mostly stayed where they were, in public housing.

  Maricela’s family was randomly assigned housing in a low-poverty neighbourhood. With her husband and their two primary-school-aged children, she moved to Cheviot Hills on the opposite side of Los Angeles. On the west side, the children did better at school, but life was lonelier. In the early years after their move, the Quintanar family would return to their old east side neighbourhood each weekend to shop, meet friends and attend church.

  Like fans awaiting a release by their favourite band, social scientists have hung on each wave of results of the Moving to Opportunity experiment.8 At first, the results seemed disappointing. Adults who moved were no more likely to be employed. Children who moved did not seem to have better school results or fewer behavioural problems. A further wave of studies showed that girls who moved were less inclined to get into trouble, while boys engaged in more risky behaviours. For a family like the Quintanars, with a son and a daughter, this would not have been an overwhelming endorsement of the program.

  The one way in which moving did seem to benefit families was in terms of health. Those who moved to low-poverty neighbourhoods were less likely to be obese, and tended to have better mental health. This should not have been surprising, given the level of danger in the communities from which families were moving. As one Baltimore boy told researchers, ‘I don’t like living around here ’cause people getting killed for nothing.’9 A Chicago mother in the control group kept her young children indoors most of the time because ‘bullets have no names’.10

  Then, in 2015, researchers at Harvard matched up results from the experiment with taxation data, allowing them to look at the earnings of children who moved to a low-poverty neighbourhood before age thirteen.11 The results were striking. Movers had earnings that were nearly one-third higher than those who’d stayed in a high-poverty neighbourhood, with the effect being roughly similar for men and women. Sustained over a lifetime, this suggests that a child who moves to a lower-poverty neighbourhood in their pre-teen years will earn $300,000 more over a lifetime. The societal gains massively outweighed the cost. Even the government came out ahead, with the participants paying more in additional taxes than the housing vouchers cost to provide.

  In the United States, large-scale social experiments date back to the 1960s. As President Lyndon Johnson announced a ‘war on poverty’, policymakers considered whether providing a guaranteed income equal to the poverty line might discourage people from working. From 1968 to 1982, experiments across nine sites randomly assigned families into treatment and control groups, and then surveyed their work patterns. The experiments showed that providing a guaranteed income reduced time spent working, but the impact was smaller than many critics expected – around two to three weeks per year.12 In coming years, the experiments helped shape reforms to the welfare system, including a massive expansion to wage subsidies in the 1990s under President Bill Clinton.13 Promising that ‘If you work, you shouldn’t be poor’, Clinton doubled the size of the Earned Income Tax Credit, pointing to the strong economic evidence that the program helped the poorest families. Today, the Earned Income Tax Credit is one of the nation’s largest anti-poverty programs. Current estimates suggest that it keeps 5 million Americans above the poverty line.14

  Another large-scale social experiment of that era was the RAND Health Insurance Experiment. Run from 1974 to 1982, researchers randomly assigned thousands of American families to health-care plans with varying levels of co-payment, ranging from 0 to 95 per cent. The study concluded that higher co-payments increased the chances that patients would discontinue treatment.15 For patients who were both poor and sick, the co-payments led to worse health outcomes, with those who suffered from high blood pressure being 10 per cent more likely to die when their plan had a high co-payment.16

  The RAND study remained the most important experimental evidence on health insurance until 2008, when an unusual situation arose in Oregon.17 The state’s government had decided to expand the number of low-income families covered by public health insurance by about 10,000. But for every available place, they had nearly nine people who wanted a spot. The government decided that the fairest way of allocating the new health-care places was to allocate them through a public lottery.

  In effect, a lottery is a randomised trial. So by tracking the health of lottery winners and losers, researchers were able to study the impact of getting health insurance. They found that winning health insurance led to greater use of health services. And, just as with the RAND experiment, Oregon researchers found that being randomly selected to get health insurance meant that people reported better physical and mental health. In a typical year, a person with health insurance had sixteen more days in which they felt physically healthy, and twenty-five more days in which they felt mentally healthy.

  Some lottery wins are more coveted than others. On 1 December 1969, CBS News suspended its regular programming to provide live coverage of the Vietnam draft lottery.18 In a glass cylinder were 366 blue balls, each labelled with one day of the year. Every young man aged eighteen to twenty-six knew that the earlier his birthday was pulled out, the more likely he was to be conscripted to the military. September 14 was the first birthday drawn out, June 8 the last. The draft spawned street protests, and protest songs, from Creedence Clearwater Revival’s ‘Fortunate Son’ to Pete Seeger’s ‘The Draft Dodger Rag’.

  The immediate impact of being conscripted was a risk of being killed, and more than 17,000 American draftees lost their lives in Vietnam. What about those who survived? Jim, whose birthday was the second one drawn out, recalls that the military ‘taught me how much I was capable of doing, and that nothing was impossible’.19 But for others, the trauma of Vietnam left lifelong scars. As one returning veteran put it: ‘A couple of years ago someone asked me if I still thought about Vietnam. I nearly laughed in their face. How do you stop thinking about it? Every day for the last thirty-eight years, I wake up with it, and go to bed with it.’20

  In a series of studies, economists have analysed the Vietnam draft lottery as though it were a randomised trial. Among those of draft age, we should not expect the life outcomes of a man born on September 14 to be different from someone born on June 8. Any significant differences reflect the impact of military service.

  For the United States, researchers found a significant negative impact on lifetime earnings, with most of the penalty coming in the 1970s and ’80s.21 Using the randomised experiment of the Vietnam draft, it turns out that veterans are also more likely to have moved interstate, and more likely to work for the government.22 Some things are unaffected by service: veterans are just as likely to be married as non-veterans, and are just as healthy.23 However, having a lower draft number (which made people more likely to be conscripted) did shape people in other ways. For example, in the early years after the war, those with lower draft numbers were more likely to be jailed for a violent crime.24 Two decades after the Vietnam War ended, surveys found that having a low draft number made men more likely to support the Democratic Party, and more likely to hold anti-war beliefs.25

  In Australia, the birthday conscription lotteries operated in a similar fashion, but the impact of being drafted to serve in Vietnam turned out to be quite different. Results from Australia’s draft lotteries suggest that those who went to Vietnam were no more likely to commit a crime, but were significantly more likely to suffer from mental health problems.26 The draf
t lottery evidence also suggests that Vietnam service had a much bigger negative impact on the earnings of Australian veterans than on that of American veterans.27

  *

  One of the big questions in social policy is how to help unemployed people find jobs. Unemployment means more than a lack of income – for many people it also represents a loss of self-esteem. As Scottish philosopher Thomas Carlyle once put it, ‘A man willing to work, and unable to find work, is perhaps the saddest sight that fortune’s inequality exhibits under this sun.’

  Economists have long known some of the factors that seem to inoculate against joblessness. Strong literacy and numeracy skills, a post-school qualification, a sunny disposition and plenty of experience are all factors that make people less likely to become unemployed. Unfortunately, none of these is easy to achieve. So what might society do to help a struggling jobseeker who lacks these qualities?

  Over recent decades, researchers have increasingly turned to randomised trials to test which programs might help unemployed people find jobs. As we’ve seen, this has often produced disappointing findings, such as the randomised evaluation of US job training which found that young people who participated in the programs earned less than those in the control group.28 It’s possible that the problem lies not with job training programs themselves, but with over-ambitious designers. After all, most of these programs lasted only weeks, and cost at most a few thousand dollars. Should we really have imagined that they would raise earnings – every year – by thousands of dollars?29

  Ron Haskins, a social policy expert at the Brookings Institution, argues that rigorous evaluations of education and employment programs typically find that around 75 per cent ‘have little or no effect’.30 Still, there are some glimmers of light. Randomised trials in Denmark, Sweden and the United States point to the value of meetings with caseworkers in helping people transition from joblessness into work.31 Part of the effect happens before the meeting, when the job-finding rate jumps by around 40 per cent. Just as we tend to brush our teeth before seeing the dentist, a meeting with a caseworker appears to improve people’s focus on finding a job.

  Fortunately, there’s also evidence that meetings themselves can help people find work, with job-finding rates increasing by 20 to 30 per cent afterwards. This perhaps reflects the fact that being an unemployed jobseeker can quickly become soul-destroying without someone to help guide you through the process. Because many unemployed workers have never been jobless before, caseworkers can provide advice on the best ways to search, how to write a résumé and how to get ready for an interview.32

  Occasionally, inexpensive interventions have a significant payoff. In 2010 and 2011 the German government posted out a cheerful blue brochure to over 10,000 people who had recently lost their jobs.33 ‘Bleiben Sie aktiv!’ (‘Stay active!’), the leaflet urged unemployed people. It discussed how the German economy had been recovering after the financial crisis, reminded people that unemployment can be bad for your physical and mental health, and suggested different ways they might look for a job. The leaflet boosted employment rates among those who received it. Each leaflet cost less than €1 to print and post, but boosted earnings among the target group by an average of €450. If you know another government intervention with a payoff ratio of 450 to 1, I want to hear about it.

  Because job search is a kind of competition, we need to make sure that programs are having a population-wide impact. To see this, imagine that there’s only one job available, and you were likely to get it over me. Now, suppose I am chosen for a program that improves my interview skills and I get the job instead. A randomised trial might show that the job search program worked because it gave me an edge. But if that came at your expense, then it’s not clear that society is any better off. In this example, the program helped its participants but did nothing to bring down the overall unemployment rate.

  We don’t worry much about this issue when we’re evaluating new medical treatments. After all, you and I aren’t competing to be healthy. But in a situation where one person’s gain might be another’s loss, a randomised trial can give a misleadingly rosy view of the effectiveness of a program.

  In 2007 the French government agreed to a novel way of answering this question.34 Rather than just randomly assigning jobseekers, they ran a huge experiment across 235 different labour markets. Not only would jobseekers be randomly chosen for intensive help, but the experiment would vary the share of people in a given town or city who were in the experiment. In some places, the experiment would cover all the jobseekers, while in others it would cover just one-quarter of people looking for a job. Unfortunately, the trial showed a bigger effect when it was covering a smaller share of the population, suggesting that a good part of the ‘gains’ were simply due to displacement effects. This disappointing finding is a useful reminder of how tricky it can be to design policies that bring down the total unemployment rate.

  In the face of rapid automation, some warn of a future labour market in which ‘Humans Need Not Apply’. In such an environment, they argue, it might make more sense to provide unconditional cash payments than to insist jobseekers keep looking for work. In January 2017 Finland set about testing this approach by randomly selecting a small group of unemployed people. Those in the study receive a ‘basic income’ of €6720 per year, which continues to be paid even if they find a job.35 The experiment, which covers 2000 people, will report its results in 2019. Advocates of a ‘universal basic income’ eagerly await the findings.

  *

  In recent decades, millions of young Americans have signed ‘virginity pledges’, promising to refrain from sex until they are married. The first such program – ‘True Love Waits’ – saw teenagers pledging ‘to be sexually abstinent from this day until the day I enter a biblical marriage relationship’. However, randomised evaluations of abstinence-only programs found no evidence that they reduced the age at which young people first had sex, or the number of sexual partners they had.36

  One possibility is that a virginity pledge is not taken particularly seriously. For example, one survey followed up five years after people had signed a virginity pledge, and found that four out of five youths denied that they had done so.37 But it’s also possible that abstinence-only programs are causing harm, by discouraging young people from planning for safe sex. After signing a virginity pledge, young people might feel it’s a bit hypocritical to carry a condom or go on the pill. Some evidence suggests that abstinence-only programs lead to an increase in unsafe sex, pregnancy and exposure to sexually transmitted diseases.38

  By contrast, other social policy programs have passed randomised trials with flying colours. Noticing the powerful impact of cigarette taxes in deterring young people from starting smoking, some researchers have experimented with financial incentives to encourage quitting.39 One firm in the United States offered employees up to $750 if they could quit smoking for a year.40 Those randomly chosen for the program were 10 percentage points more likely to quit. Because smokers have worse health and take more breaks during the day, they tend to be less productive than non-smokers.41 One study suggested that the productivity gap between smokers and non-smokers exceeds $2000 a year.42 This suggests that paying employees $750 to kick the habit is a great deal for the company – even ignoring the benefits to workers and their families.

  Similarly strong results could be seen in a randomised trial in the Philippines, in which smokers were invited to deposit money into a savings account.43 After six months they took a urine test for nicotine. Those who passed got their money back. Those who failed saw their savings donated to charity. The average deposit was significant – enough to buy cigarettes for half a year. Not surprisingly, participants wanted the money back, and this boosted their odds of quitting by 3 to 6 percentage points.

  Mark Twain once said, ‘Quitting smoking is easy. I’ve done it a thousand times.’ The results from randomised trials suggest that perhaps Twain – who was often looking for ways to earn money – only neede
d the right financial incentive.44

  We saw at the start of the chapter that programs for the long-term homeless don’t necessarily move them into employment. But targeted assistance can achieve more modest goals. In New York, one experiment focused on people being released from psychiatric hospitals: strengthening their ties to friends and family, and providing emotional support.45 Those randomly chosen to get support were five times less likely to be homeless. Timely assistance kept them off the streets.

  Smart social policies can make a difference, but when it comes to building opportunity, plenty of people have argued that education is crucial. In a moment, I will delve into the world of educational experiments. But before that, let’s take a pause from the research findings to look at some of the stories of randomistas who have shaped the field.

  4

  THE PIONEERS OF RANDOMISATION

  Charles Saunders Peirce had a rare nerve condition in which a slight touch of the face can trigger an agonising sensation as powerful as electric shock. According to his biographer, the pain caused him to be ‘aloof, cold, depressed, extremely suspicious, impatient of the slightest crossing, and subject to violent outbursts of temper’.1 To control it, he turned to drugs, including ether, morphine and cocaine. Peirce was largely home-schooled – by his father, a mathematics professor at Harvard. When Peirce himself arrived at Harvard, he often found himself bored in class, and graduated near the bottom of his year. Yet in 1885 Peirce published one of the first randomised experiments in the social sciences.2 The goal was to test the accuracy of our sense of touch. In particular, the experiment aimed to assess how good we are at comparing physical weights. If I put a blindfold on you and put two different weights in your hands, I’ll bet you could easily distinguish one object that weighs 50 per cent more than another. But could you tell the difference if the discrepancy in weights was just 10 per cent? Or 1 per cent?