Randomistas

by Andrew Leigh

  Many of MDRC’s evaluations showed that programs worked, but they rarely worked miracles.24 Sometimes the successes were confined to particular groups of participants, or to certain welfare offices. The experiments often overturned the preconceptions of the social work community. In Louisville, counsellors were confident that they knew which clients were job-ready. Gueron’s random assignment study proved that it wasn’t possible to predict success in advance: the job market often threw up surprises.

  Taking over as MDRC president in 1986, Gueron became better at explaining to people who ran social programs why random assignment was fair. In one instance, a county commissioner continued random assignment after the research ended, seeing it as a just and unbiased way of choosing who would be assisted.25

  Others weren’t so warm. In 1990 Florida legislator Ben Graber tried to shut down MDRC’s evaluation of ‘Project Independence’, an employment program. Random assignment, Graber said, was treating welfare recipients like ‘guinea pigs’. He foreshadowed legislation that would ban the use of control groups. The media was quick to pick up the story. The Miami Herald editorialised that randomisation was ‘misguided, wasteful, cold-hearted, and just plain dumb’. The St. Petersburg Times editorial called it ‘a cruel joke’ which would spend millions of dollars to deny mothers with small children the job training they deserved. Within weeks, politicians in other states were announcing their opposition to randomised trials. The Washington Times predicted: ‘Public protest precipitates end of welfare experimentation’.26

  Testifying in Florida, Gueron focused on the fact that the program had never been properly evaluated. ‘If we had a readily available wonder drug to help people be self-sufficient and off of welfare, I’m sure there isn’t a person in this room who would not use it. If Project Independence is that, I assume you would put more money into it. Since we don’t know that and the funds are not there, you would be wise to get the answers first.’27 The legislature approved the trial. A few years later, the evaluation results were in: the employment program saved the taxpayer about as much as it cost to run. It wasn’t a wonder drug, but it was worth continuing. The Florida press had compared Gueron’s team to scientists pulling the legs off spiders, but they had fought back, and managed to produce the definitive evaluation of Project Independence.

  Before the use of randomised evaluation, debates over social programs often featured conflicting studies, using a tangle of approaches. When the experts diverged, policymakers and the public were torn. As economist Henry Aaron remarked, ‘What is an ordinary member of the tribe to do when the witch doctors disagree?’28 In the Florida case, the official in charge of Project Independence came to Gueron after commissioning three non-randomised studies, all of which used different methodologies and arrived at different conclusions. ‘The only way to get out of the pickle of these duelling unprovable things,’ he decided, ‘was to get an evaluation of unquestioned quality.’29 To Gueron, randomised evaluation was a powerful communications tool because of its simplicity. ‘Anyone could understand the basics of the analysis. There was no fancy statistical footwork.’30

  In the late 1960s, Gueron recalls, ‘researchers knew in theory the power of random assignment. They just didn’t believe it would be useful to evaluate real-world social programs and address important policy questions.’31 But over her career Gueron successfully deployed policy experiments in courtrooms, schools, community colleges, job training centres and community organisations. Over time, she ceased warning her colleagues that ‘no one wants to be in a random assignment study’.32 MDRC was turned down more often than accepted, but the popularity of social experiments steadily grew. They were straightforward, and made it harder to hide failure.

  ‘The key thing about random assignment,’ Gueron concludes, ‘is that it is the epitome of transparency. You flip a coin. You create two groups or more. You calculate average outcomes. And you subtract . . . That power is to be treasured, and contrasted with many prior and other evaluations that have led to disputes among experts about methodology, which is the death knell for research having an impact on policy.’33 The chief lesson she draws from her career: ‘Fighting for random assignment is worth it.’

  5

  LEARNING HOW TO TEACH

  Oscar the Grouch gives children permission to feel sad. Big Bird questions everything. Mr. Snuffleupagus is the imaginary friend. Count von Count loves mathematics. Grover embodies self-confidence. Ernie delights in practical jokes. Bert has an utterly different personality to Ernie, but is his best friend nonetheless. Kermit the Frog is always a gentleman.

  In 1967, Joan Cooney began to plan a television show unlike any other: a collaboration between creative designers and child science experts. At a time when children were watching shows like Looney Tunes and The Flintstones, this would be a television program that followed its own academic curriculum. Most unusually, the Children’s Television Workshop would use evidence to shape the show. The idea of combining research with television production was, Cooney says, ‘positively heretical’.1 And so it was that an unorthodox team of designers and social scientists created Sesame Street.

  In its first year, Sesame Street was evaluated in a randomised trial, which compared a treatment group (children who were encouraged to watch the program) with a regular control group. Unfortunately, the researchers hadn’t reckoned on the show’s popularity. With more than one-third of American children tuning in to each episode, there wasn’t much difference in viewing rates between the two groups.2

  So the next year, researchers took a different approach – focusing on cities where Sesame Street was only available on cable and then randomly providing cable television to a subset of low-income households, whose children were encouraged to watch Big Bird and friends. This time, there was a big difference in viewing rates between the control group (without Sesame Street) and the treatment group – and a significant difference in vocabulary.3 Children who watched Sesame Street had the same cognitive skills as non-viewers who were a year older.

  In the past half-century, over a thousand research studies have been conducted on Sesame Street, many of them feeding back into the show’s development.4 In one experiment, the show’s designers wanted to find out how best to teach preschoolers the functions of their eyes, nose and mouth. Groups of young children were randomly assigned to watch either a test video featuring Grover interacting with a little girl named Chelsea, or a video with Elmo pointing out body parts on the Mona Lisa painting. When tested afterwards, children who had been allocated to watch the video with Grover had a better grasp of how their body parts worked than children who watched the video with Elmo. The researchers concluded that the painting was too abstract a teaching tool, and that children learned body parts best in a segment featuring both a Muppet and a human actor.

  In teaching children about the concepts of animate and inanimate objects, a randomised study showed that preschoolers became confused when the show included both plants and animals. Children could understand why a chicken was in a different category from a rock – but adding a tree created too much confusion. After testing different variants of the ‘What’s Alive?’ segment, the show designers removed the tree and aired a segment simply comparing animals with inanimate objects.

  Another question was how many letters of the alphabet should be taught in each episode. Sesame Street designers randomly assigned preschoolers to watch episodes with two ‘Letters of the Day’ or with one ‘Letter of the Day’. They found that doubling the letters wasn’t an effective way to teach. Asked afterwards to correctly identify letters, children who watched episodes with two letters were less likely to know either of them than children who had watched an episode with just one letter.

  Research even determined which subjects went to air. A pilot program about the death of storekeeper Mr Hooper showed that it gave children a better understanding of death, without adverse reactions. But a pilot about the divorce of Mr Snuffleupagus’s parents left some children in the test audience
with the mistaken impression that all parental arguments invariably lead to divorce. Sesame Street aired the show about death, but the program about divorce was never broadcast.

  *

  How we understand childhood is inextricably linked to the question of whether it’s worth investing early in helping children. Indeed, we often put the spotlight on what young children are unable to do. They can’t feed themselves, walk or kick a ball, and they make lousy dinner party guests. The word ‘infant’ derives from the Latin adjective infans, meaning ‘not speaking’. Inadvertently, we see children as defective adults. We think the role of parents and educators is to correct their mistakes, keep them out of trouble and help them grow up to be perfect, like mum and dad (or so we tell ourselves!).

  Psychologist Alison Gopnik advocates a different view. If a family were a corporation, she says, children would be the research and development division, while adults would be the production and marketing departments. ‘They [children] think up a million new ideas, mostly useless, and we take the three or four good ones and make them real.’5 From this perspective, creating environments where children can play creatively is vital to ensuring we thrive as a species. It also shapes how we think about children from disadvantaged backgrounds, and whether their life chances are predetermined or malleable.

  In 1958 another psychologist, David Weikart, took up the job of being director of special education in Ypsilanti, Michigan.6 At that time, schools were segregated, and all the African-American students in the town attended one primary school – the Perry School. Weikart noticed that the school was run down. Instead of a playground, it had a field filled with thistles. Many of the African-American students ended up repeating grades, entering special education or leaving school early.

  Yet when Weikart gave a presentation to school principals about these problems, users responded defensively. One sat with arms tightly folded; others stood by the window smoking; a few left the room. When he pressed them to act, they said there was nothing they could do. Black students were just born that way. So Weikart came up with an alternative solution: ‘Because I couldn’t change the schools . . . well, obviously you do it before school.’

  In the late 1950s the only institutions that looked anything like preschools were nursery schools, focused purely on play. By contrast, Weikart was interested in the work of psychologists such as Jean Piaget, which suggested that young children’s minds are actively developing from the moment they are born. But when it came to early intervention, Weikart noted, ‘There was no evidence that it would be helpful. There wasn’t data.’ So he decided to put Piaget’s theories to their first rigorous test.

  In 1962 the Perry Preschool opened, for children aged three and four. Twenty-eight families had expressed interest in enrolling their children. From these, the researchers had chosen thirteen to attend preschool, while fifteen remained as a control group.7 The selection was random – literally made by the toss of a coin. Over the next four years, the experiment would grow to 123 children (fifty-eight who attended, and a control group of sixty-five).

  Former Perry Preschool teacher Evelyn Moore remembers how the program pushed back against the prevailing wisdom that a child’s intelligence was fixed, and that many of the children in the community were ‘retarded’. She saw something different – these children knew the names of baseball players. They recalled the words to songs. And their parents had hope. When Moore visited the families at home, she saw that almost all had pictures on the wall of two men – John F. Kennedy and Martin Luther King.

  The preschool curriculum was highly verbal. Children were taught to draw, to make up stories and to complete increasingly complicated puzzles. Teachers asked open-ended questions, in a teaching technique known as ‘verbal bombardment’.8 They visited a farm, a fire station and an apple orchard, where they picked apples and cooked them into apple sauce. Months later, in winter, they went back to the orchard to see the seasonal change. When Evelyn Moore asked the children where the apples had gone, one child reflexively replied, ‘Teacher, I didn’t take ’em.’

  Decades later, one participant, ‘David’, recalled how Perry Preschool wove education throughout the daily experience: ‘even during the playtime there seemed to be a learning component to it. I understand it now as relationship building, playing games with others, getting used to interacting with others.’

  The Perry Preschool program lasted only two years, but over the coming decades researchers tracked the outcomes for those who had participated, and for the randomly selected control group. In the early years they found that those who had attended preschool did better on IQ tests – but this effect faded over time.9 Then in the teenage years they began to notice other impacts. Seventy-seven per cent of preschool participants finished Year 12, compared with 60 per cent of those in the control group. By the time they were in their twenties, those who had been to preschool were more likely to own a car, own a home and have a steady job. They were also less likely to use drugs and less likely to be on welfare. By age forty, 28 per cent of those in the preschool group had been to jail, compared with 52 per cent of the control group.

  The leading economic analysis of the program estimates that for every $1 spent on Perry Preschool, the community gained between $7 and $12.10 By far the biggest benefit came from reduced crime, showing that if you target early intervention at people with a fifty-fifty chance of going to prison, you can change the lives of participants at a reasonable cost to the broader community.

  In the decades since, randomised evaluations of early childhood programs for extremely disadvantaged children have continued to show benefits.11 From 1972 to 1977, 111 infants in North Carolina participated in the Abecedarian experiment, which provided five days a week of care for children from four months of age all the way until they entered primary school. When they grew up, participants were more likely to attend university, more likely to be employed as adults, and tended to wait longer before having children of their own.12 By the time the participants were in their mid-thirties, it had been three decades since the program ended. And yet those who had been part of the Abecedarian Project had significantly lower blood pressure.13

  Other randomised trials to help young children have focused on parenting strategies. ‘Triple P’, a positive parenting program designed at the University of Queensland, works with parents over about eight sessions to build up skills in praising children, creating engaging activities, setting rules and managing misbehaviour. Randomised trials have shown Triple P to be effective at making parents feel more confident in their parenting skills, and at reducing the amount of misbehaviour that parents observe.14

  Seeing these results in the mainstream community, a team of education researchers in Brisbane worked with Indigenous social workers to develop a culturally appropriate version of Triple P, and then tested it via a randomised trial.15 The experiment involved fifty-one families, the sort of sample size that often makes it impossible to discern true policy impacts from background statistical noise. However, in this case the effect of Triple P was so large that it led to a statistically significant drop in the share of children who showed behavioural problems. Psychologists administered a test that asked about thirty-six problematic behaviours. As a result of Triple P, children reported six fewer of these behaviours – moving the average child in the sample out of the ‘clinical range’ and into the ‘normal range’.

  In Ireland, which has one of the highest rates of child poverty in the advanced world, a twelve-session positive parenting program called the Incredible Years Basic Parenting Programme has shown similarly powerful results in children aged three to seven.16 Targeted at some of the most disadvantaged children in Ireland, the early findings of the randomised trial suggest that intervening in the ‘incredible years’ could ultimately have incredible results for the community, reducing crime, welfare spending and health-care bills.17

  Some parenting interventions begin when the children are just infants, with home visits by nurses in the firs
t months of a child’s life.18 These programs typically target children who might be ‘at risk’: because the parents are poor, because the babies had low birth weight, or because social service agencies are concerned about the family’s wellbeing. Nurses provide counselling, offer advice on getting babies to sleep and remind parents of the importance of talking and singing to their newborn. Randomised evaluations have found that home visits from nurses lead to better parenting and improvements in child cognition. There is also evidence that nurse home visits reduce the number of women who suffer from physical or sexual violence at the hands of their partners.19

  But a careful synthesis of the evidence also pointed to the value of randomised trials in getting a realistic measurement of the effects. It found that non-randomised studies tended to overestimate the benefits of nurse home visits by a factor of three to six.20 It’s hard to be sure precisely what went wrong with the non-randomised studies, but one possibility is that they compared parents who asked for a home visit with those who did not request one. If parents who wanted a nurse to come to their home were a little more motivated, then this would likely skew the result in favour of home visits.

  Neuroscience has provided a powerful motivation for early intervention. As early childhood researcher Dana Suskind puts it, our liver, lungs and heart work perfectly from day one. Only the brain is partially developed at birth.21 And yet there is also a risk that neuroscience will be misused – either to justify any early intervention, or to suggest that disadvantage in the early years causes irreversible damage. Some people mistakenly jump from the fact that the brain is four-fifths of its full size by age three to conclude that all early years programs are great value for money. One shocking brain scan showing the underdeveloped brain of a three-year-old pops up frequently in PowerPoint presentations and TED talks by advocates, but no one seems to know where the image comes from, let alone the circumstances of the child’s upbringing. In Europe, the ‘1001 critical days’ movement has argued that this period determines how a child functions throughout life – sometimes going so far as to claim that ‘age two is too late’.22 The movement would do better to focus on rigorously analysing what works, rather than alleging that it’s ‘game over’ once a child becomes a toddler.