by Andrew Leigh
In Melbourne, half a century after David Weikart set up the Perry Preschool, a group of researchers are applying a scientific approach to understanding how to design an effective early childhood program for children at risk. The Early Years Education Program is Australia’s first randomised trial of high-quality early learning for extremely disadvantaged children.23
Walking into the early years centre in West Heidelberg, I’m struck immediately by how homely it feels, with soft lighting, rugs and couches. The play area looks fresh, and there are chickens and guinea pigs in cages. Inside, the educators exude calm and confidence. Most have more than a decade’s experience in early childhood. At the youngest ages, there are only three children for every adult.
Like a hospital intensive care unit, the aim of the centre is to repair trauma. Severe challenges like drug abuse and family breakdown have led to an atmosphere of ‘toxic stress’ in the lives of the infants and toddlers who attend. A four-year-old boy, ‘Will’, has been excluded from two childcare services because of biting, swearing, spitting and urinating in the playroom.24 Will’s mother endured three violent relationships, and Will himself was sexually abused at the age of three by one of his mother’s partners. His mother regularly smacks him and shouts at him, and Will still wears nappies because he is too frightened to use the toilet.
Left in this situation, Will could easily end up in child protection, on welfare or in jail. Over the course of his life, he might experience extreme hardship, as well as costing the community hundreds of thousands of dollars. And yet the West Heidelberg centre isn’t cheap either. Will benefits from highly qualified educators, small group sizes and a kitchen that provides him with two meals a day. The early signs are good: Will has been put in charge of caring for the chickens and collecting their eggs. But only a rigorous analysis will help us know if the benefits to Will and the other children at the centre justify the expense.
Talking with parents involved in the Early Years Education Program, I’m struck by the tension that they felt during the randomisation process.25 A mother told me she held her breath as the news came through on the telephone. Parents are told when they apply that their odds of getting a place are 50/50. To some, it feels like a lottery, but places in the centre are limited and families who are not chosen are still able to access regular childcare. As one researcher observes to me, ‘A 50 per cent chance is better than many of these families get in many other parts of their lives.’
Without randomisation, there’s invariably the risk that those who attend the centre will be the children with the most motivated parents, whose children might have had better outcomes regardless. Because the researchers are confident that they have a credible comparison group, they hope to follow both groups for the next fifty years: to understand whether a great early childhood education really does transform lives.
*
Among education researchers, randomised evaluations have traditionally been viewed with suspicion. There is no shortage of discontent about the current state of knowledge, and much has been spent on educational evaluations. But as one analysis pointed out a decade ago, ‘not much has been learned about what works’.26 Prominent education researchers have tended to reject randomised trials, wielding arguments that the world is too complex, control groups are unethical, or randomised trials are politically unfeasible.27
Making schools work better is a truly complicated task. From their first day of class to Year 12, students spend around 16,000 hours at school. But that’s only a small fraction of their waking hours, so it’s perhaps not surprising that around half the variation in student performance is determined by families, not schools.28 Teachers have always had to deal with young people’s backchat, indolence and hormones. Today, they must also hold the attention of children who are avid users of smartphones, tablets and gaming consoles.
Schools, too, are facing challenges in attracting and retaining staff. Half a century ago, schools were one of the main employers of working women, who faced much worse gender discrimination in other occupations. Now, high-achieving women have promising career options in business, medicine and law. Consequently, the academic standards of new teachers have slipped backwards.
The results are showing up in test scores. Over the past century, intelligence tests administered across the population have shown a steady increase, decade on decade. But it now looks like the ‘Flynn effect’ – named after New Zealand social scientist James Flynn – may be driven mostly by the fact that we are getting more education rather than because our schools are improving. Over the past two decades, in the Programme for International Student Assessment, the OECD has administered standardised tests to a sample of 15-year-olds. In many advanced nations, test results are getting worse. Average test scores in advanced countries have fallen in mathematics, reading and science.29 How do we turn these trends around?
Randomised trials of school-based programs can produce some surprises. In recent years, the US Department of Education devoted over a billion dollars annually to an after-school program known as the 21st Century Community Learning Center initiative.30 Children attend the centres for up to four hours of after-school programs, which can involve everything from tutoring to drama to sports. An early evaluation of the program drew positive conclusions about its effectiveness. Surveying teachers, researchers found that children who attended the centres had made improvement over the year in their academic performance, motivation, attentiveness and classroom behaviour.31
This sounds promising, until you stop and think for a moment about the counterfactual. In effect, the researchers were assuming that from one year to the next, children make no improvement in their understanding of the subject matter or ability to learn. They were attributing any improvement in how students think or behave to the after-school program. This meant that the evaluators were putting a heavy thumb on the scales in favour of the program.
Then a team of economists at Mathematica, a policy research organisation that specialises in randomised trials, released the results of their evaluation. Rather than comparing students to how they were a year ago, the Mathematica evaluation randomly assigned primary school students to attend an after-school program or to the control group (which generally meant being at home with a parent or relative).
The results were starkly different from the earlier evaluation. Children who attended after-school programs were significantly more likely to misbehave at school – engaging in behaviour that led to detention, to their parents being called to school, and to discipline problems.32 Attending an after-school program raised the chances of a child being suspended from 8 per cent to 12 per cent: perhaps because going to an after-school centre raised the odds that children would fall in with the wrong crowd. Ultimately, there was no evidence that the after-school program improved academic outcomes, but plenty of evidence that it worsened behaviour.33 A program costing American taxpayers over a billion dollars a year was failing to improve student learning and creating disciplinary problems.
Unfortunately, the story does not have a happy ending. A group of advocates, the most prominent among them the former Californian governor Arnold Schwarzenegger, have successfully lobbied to maintain federal funding for 21st Century Community Learning Centers. Schwarzenegger felt no obligation to discuss the data or explain why he thought the Mathematica study was flawed. Instead, he airily argued, ‘It would be a mistake, let me repeat, a big mistake, to use that study as justification to reduce current funding levels for after-school programs.’34 The US Congress decided to keep funding the program. Congress might otherwise have chosen to spend the money in ways that have been proven to help low-income children. For example, a billion dollars could provide home visits from nurses for 88,000 first-time mothers, high-quality early childhood programs to 96,000 preschoolers, intensive reading support to 295,000 third-graders, or an evidence-based program to reduce teen pregnancy to 1.3 million teens.35
In the United Kingdom, the Education Endowment Foundation (EEF) has commi
ssioned over a hundred evaluations, many of them randomised, to test what works in the classroom. Among those randomised evaluations that produced positive results are personal academic coaching, individual reading assistance, a Singaporean-designed mathematics teaching program, and a philosophy-based intervention encouraging students to become more engaged in classroom discussion.36
Because the EEF is running a large number of randomised trials, its evaluation experts are also working to ensure that researchers can compare the size of the results. Just as when buying a car, the right question isn’t ‘How good is it?’ but ‘What do I get for my money?’ Resources will always be scarce in education, so knowing which programs are the most efficient is vital.
One way that the EEF compares programs is by looking at how much it costs to improve a pupil’s performance by the equivalent of one month’s learning.37 Even among effective programs, they found wide divergence. To get a one-month improvement for one student, personal academic coaching cost £280, individual reading assistance cost £209, the mathematics teaching program cost £60, and the philosophy-based intervention cost £8.38 So while all the programs ‘worked’, some were a whopping thirty-five times more cost-effective than others.
In some cases, the EEF trialled programs that sounded promising, but failed to deliver. The Chatterbooks program was created for children who were falling behind in English.39 Hosted by libraries on a Saturday morning and led by trained reading instructors the program gave primary school students a chance to read and discuss a new children’s book. Chatterbooks is the kind of program that warms the cockles of your heart. Alas, a randomised trial found that it produced zero improvement in reading abilities.
Another EEF trial tested the claim that learning music makes you smarter. One former president of the National Association for Music Education insisted: ‘Music enhances knowledge in the areas of mathematics, science, geography, history, foreign language, physical education, and vocational training.’40 Most studies of this question have compared those who choose to study a musical instrument with those who do not. But children who take up the violin at age three might already be different – more driven, more cognitively gifted or with pushier parents – than those who do not. In the EEF study, 900 students were randomly assigned either to music or drama classes, and then tested for literacy and numeracy.41 The researchers found no difference between the two groups; suggesting either that learning music isn’t as good for your brain as we’d thought, or that drama lessons are equally beneficial.
Across the Atlantic, even fewer educational interventions have been found to be supported by rigorous evidence. In 2002 the United States established the What Works Clearinghouse, a federal body responsible for assessing the scientific evidence on which education interventions are most effective – looking at everything from science teaching to assisting students with disabilities. In its first decade, nine in ten of the randomised trials commissioned by the What Works Clearinghouse failed to produce positive effects.42 Wags began calling it the ‘Nothing Works Clearinghouse’.43
Over the years since, the Clearinghouse has managed to garner bipartisan support. As well as commissioning new studies, the body is also responsible for sifting through the available evidence – looking at what the best evaluations conclude. To get the highest rating, studies must use random assignment.44 Given the decentralised nature of US school education, the What Works Clearinghouse serves as a resource for educators and parents alike.
While many new programs don’t work, some have produced significant impacts. In one experiment, the Bill & Melinda Gates Foundation conducted a randomised trial of coaching programs for teachers. Each month, teachers sent videos of their lessons to an expert coach, who worked with them to eliminate bad habits and try new techniques. By the end of the year, teachers in the coaching program had seen gains in their classroom equivalent to several additional months of learning.45
With technology proliferating in schools, randomised trials are exploring whether hardware and software can help children learn faster. Giving free computers to schoolchildren tends not to have much of an impact on students’ literacy and numeracy scores.46 But experiments evaluating online learning tools have found sizable impacts, particularly for mathematics.47 By turning problems into games, breaking down skills into digestible chunks and providing rapid feedback, applications such as ASSISTments and SimCalc have boosted achievement in arithmetic, algebra and calculus. Randomised trials have backed some unconventional strategies to raise school performance. In Israel, a randomised trial in forty low-achieving schools saw students offered a $1500 payment for passing their final exams. The payment raised school completion rates by more than one-third.48 In the UK, weekly text messages to parents about the dates of upcoming tests and what students were learning that week boosted the performance of high schoolers.49 Relative to a randomised control group, the text messages (costing £6 per pupil) improved mathematics performance by a month of additional learning per year. Another cost-effective intervention sent letters to parents whose children were missing a lot of school (on average, seventeen days per year).50 Because parents underestimated their children’s absences, simply telling them the statistics on how many days their children missed had the effect of cutting absences by one-tenth.
Massive impacts have also been observed in a randomised evaluation of the Promise Academy, an unusual school in New York’s Harlem district. Outcomes for young people in Harlem were dreadful: a study once found that life expectancy for young men born in Harlem was lower than for those born in Bangladesh.51 Cocaine, guns, unemployment and family breakdown created an environment where disadvantage was perpetuated from one generation to the next. Founded in 2004, the Promise Academy has an extended school day, with classes running from 8 am to 4 pm, and after-school activities often continuing until 7 pm. There are remedial classes on Saturdays, and the summer break is shorter than in most schools. The result is that students spend nearly 50 per cent more time in school than the typical child. The school operates on a ‘no excuses’ model, emphasising grit and perseverance. It is assumed that every child will go on to university. Both students and teachers are heavily monitored, with a strong focus on test score gains.
The Promise Academy is one of 6900 public charter schools in the United States. When such schools have more applicants than places, they assign spots based on a lottery. Like a Powerball draw or the Vietnam draft, these charter school lotteries are carried out publicly. A few use computer algorithms. Others write students’ names on pieces of paper and draw them out of a box. In some cases, schools borrow the same equipment used by gambling lotteries, assigning students a number and then drawing the balls out of a cage. The documentary Waiting for Superman shows the selection process for two Harlem schools, where there are twenty applicants for every lottery place. As students are selected, we see their parents screaming for joy. Afterwards, we see the tears on the faces of unsuccessful parents. Those who miss out include Bianca, a kindergarten student whose mother works multiple jobs in the hope of Bianca getting to college; and Francisco, a first-grader who is struggling with his reading.
From the ecstasy and heartbreak of the lottery comes a clear comparison. Researchers followed students like Bianca and Francisco, and compared them with students who had won a place in the Promise Academy. They found that the school had a massive impact on performance. Across the United States, the average black high school student is two to four years behind his or her white counterparts. Students who won a lottery to attend the Promise Academy improved their performance by enough to close the black–white test score gap.52 As lead researcher Roland Fryer points out, this overturns the fatalistic view that poverty is entrenched, and schools are incapable of making a transformational difference. He claims that the achievements of the Harlem Children’s Zone are ‘the equivalent of curing cancer for these kids’. ‘It’s amazing,’ he muses. It should be celebrated.’53 In a nation where racial gaps persist everywhere from home ownership
to life expectancy, programs like this one offer the potential for fulfilling the founding promise that ‘all men are created equal’.
Passion for social reform can go along with rigorous assessment of interventions. The Thirty Million Words Initiative is named after a 1995 study which suggested that disadvantaged children might hear 30 million fewer words by their fourth birthday than children from affluent backgrounds.54 The institute’s founders thought that one way of overcoming that gap might be to provide parents with feedback about how well they were doing in chatting with their children. When the study was carried out over two decades ago, the only way of counting words was to have a researcher in the room. But today voice recognition devices can do the job. For around $400, a digital device called LENA (for ‘language environment analysis’) sits in a child’s pocket and counts how many words parents and children say to one another. So the Thirty Million Words Initiative is carrying out a randomised trial to see whether giving parents daily data on how much they have been speaking with their children will encourage them to do more of it. So far, the results are promising.55