by Ben Goldacre
EVIDENCE-BASED POLICY
I’d Expect This from UKIP, or the Daily Mail. Not from a Government Leaflet
Guardian, 15 April 2011
The government has issued a new leaflet aiming to justify the latest round of redisorganisation in the NHS. This leaflet is called ‘Working Together For A Stronger NHS’. It was produced by Number 10, it appears on the Department of Health website, and many of the figures it contains are misleading, out of date, or factually incorrect.
The leaflet begins, like much pseudoscience, with some uncontroversial truths: the number of people aged over eighty-five will double, and the cost of drugs is rising. This is all true.
Then comes the trouble. In large letters, alone on one entire page, you see: ‘If the NHS was performing at truly world-class levels we would save an extra 5,000 lives from cancer every year.’ The reference for this is a paper in the British Journal of Cancer called ‘What if Cancer Survival in Britain were the Same as in Europe: How Many Deaths are Avoidable?’
This study does not aim to predict the future: in fact, it looks at data from 1985 to 1999 – seriously – which is a very long time ago. It finds that if we’d had the same cancer survival rates – more than twenty years ago, in the eighties and nineties – as the average in the EU, then we’d have had 7,000 fewer deaths per year. Not 5,000 fewer. To put the big number in context, by this study’s calculation 6–7 per cent of UK cancer deaths in the 1990s were avoidable. Since then we’ve seen the massive 2000 NHS Cancer Plan, a new decade, and a new century. This paper says nothing about the number of lives we ‘would save’ each year by 2011, and citing it in that context is entirely misleading.
The next interesting figure misleads about a trend (we’ve seen this a lot from health ministers recently) and attempts to take the credit for a long-standing change. The leaflet says: ‘Since May 2010 the NHS has gained 2,550 more doctors and has 3,000 fewer managers.’ This is correct: full-time equivalent figures (my favourite) from NHS workforce data show 97,720 doctors in May 2010, and 100,197 in December. But NHS Information Centre figures show that between 1999 and 2009 the total number of doctors increased from 88,693 to 132,683, GPs from 28,354 to 36,085, and consultants from 21,410 to 34,654. Doctors take a while to grow, and they’ve been growing in number for a good long while.
Then we have choice: ‘95 per cent want more choice over their healthcare’. The source given is the twenty-fifth British Social Attitudes Survey. Interestingly, the government has just announced that it’s going to stop funding the health questions in the British Social Attitudes Survey, so this valuable resource won’t be around for long. The data was collected in 2007, it’s free to download, and if you do so you’ll see it didn’t ask about ‘more choice’. Question 583 asks how much choice you think NHS patients currently have (‘a lot’, ‘a little’, and so on), and Question 584 asks how much choice you think they should have.
How those responses were aggregated to get 95 per cent of people in favour of ‘more choice’ – a key justification for reform – is a mystery: many people will have said they have ‘a little’ choice, and that they should have ‘a little’ choice (we can’t see how many from the aggregated data in tables of course; we need respondent-level data, because that’s the only way we can link together an individual’s responses to a sequence of questions). I asked the government how it produced its figure, since BSA25 doesn’t have data on the question ‘Would you like more choice?’ I was told the source was table 3.1 in Chapter 3 of a book that costs £52, called Do People Want More Choice and Diversity of Provision in Public Services.
I got that book: it’s the same old BSA25 data. It doesn’t contain anything on ‘more choice’, and they got the title wrong: it’s not Do People Want More Choice … It’s Do People Want Choice … which shows how misleading they’re being, and wasted me a lot of time in the library. None of this should be difficult. The facts in this plainly political pamphlet should be clean, correct, transparent and justified. As the government defies all reason by claiming that NHS staff support its reforms, we can only fear the results of its new ‘listening exercise’.
Andrew Lansley and His Imaginary Evidence
Guardian, 5 February 2011
I have never heard one politician use the word ‘evidence’ so persistently, and so misleadingly, as Health Secretary Andrew Lansley defending his NHS reforms. Since he repeatedly claims that the evidence supports his plan, let’s skim through what we can find on whether GP consortiums work, the benefits of competition, and the failures of the NHS.
Are GP consortiums better than Primary Care Trusts for commissioning? There have been fifteen major reorganisations of the NHS in thirty years. We’ve had GP fundholders, GP multifunds, primary care groups, primary care trusts, family practitioner committees, purchasing consortiums, and more. After all this change, lots of data should have been gathered on the impact of specific strategies.
In reality, few of them were properly studied. Here are four papers on GP fundholding, which is broadly similar to Lansley’s GP consortiums. Kay in 2002 found it was introduced and then abolished without any evidence of its effects. In 2006 Greener and Mannion found a mix of good and bad but no evidence that it improved patient care. In 1995 Coulter found nothing but gaps in the evidence and no evidence of any improvement in efficiency, responsiveness, or quality. Petchley found there was insufficient data to make any judgement. Lansley says he is following the evidence. I see no evidence to follow here.
Next, competition. Andrew Lansley has repeatedly denied that he is introducing competition on price. This is disturbing behaviour: his Bill explicitly introduces price-based competition – it’s in paragraph 5:43 of his ‘NHS Operating Framework’.
Does variable-price competition work in healthcare markets? Working from first principles, markets for healthcare in which people compete on price as well as quality might be expected to produce lower-quality healthcare, because prices are easy to measure, while quality, in healthcare, is surprisingly difficult to measure: so quality suffers.
It’s hard to research this kind of thing, but even the evidence on fixed-price competition – where you compete on quality – is mixed. There are various ways to assess it. Often people choose an outcome – like the number of people who survive a heart attack – and compare this outcome in areas of more intense or less intense competition. Sometimes competition makes things worse, sometimes better.
For variable-price competition, which is what we’re facing, things don’t look good. Its introduction in New Jersey in the 1990s was associated with a worsening death rate from heart attacks, while in the UK, stopping variable-price competition was associated with improvement. These aren’t clean, easy interventions to assess, but despite his using the word repeatedly, again the ‘evidence’ does not support Lansley here.
Lastly, there is the justification for reform. Both Lansley and David Cameron are – rather shamefully – overstating our mortality figures, in order to claim that the NHS is failing. Everyone wants more improvement, but money or a structural change does not produce an immediate and visible reduction in mortality from one thing, so it’s hard to use these figures to pin blame or credit on anyone; interventions take time to have an impact, especially on things that kill you slowly; and NHS treatment isn’t the only factor affecting how many people die of something. But since you’re interested, to take just two things: mortality from cancer has fallen every year since 1995, and heart attack deaths have halved since 1997.
The government claims that our rate of death from heart attacks is double that in France, even though we spend the same on health. Health economist John Appleby instantly debunked this claim in the BMJ, and his piece will become a citation classic. From static 2006 figures in isolation, the government is right. But the trajectory of improvement in the UK is so phenomenal that if the straight line continues – as it has done for thirty years – we will be better than France within a year.
I’m not in favour of – or ag
ainst – anything here: genuinely, all health-service administrative models baffle and bore me equally. But when Andrew Lansley says all the evidence supports his interventions, as he has done repeatedly, he is simply wrong. His wrongness is not a matter of opinion, it is a fact, and his pretence at data-driven faux neutrality is not just irritating, it’s also hard to admire. There’s no need to hide behind a cloak of scientific authority, murmuring the word ‘evidence’ into microphones. If your reforms are a matter of ideology, legacy, whim and faith, then like many of your predecessors you could simply say so, and leave ‘evidence’ to people who mean it.
Why Is Evidence So Hard for Politicians?
Guardian, 12 February 2011
One thing you hope for, with politicians, is that they won’t make the same mistakes over and over again.
Last week we saw that the government has overstated the problems in the NHS by using dodgy figures (to be precise, it used misleading static figures instead of time trends). We also saw that Andrew Lansley’s frequently repeated claim – that his reforms are justified by evidence – is untrue: the evidence doesn’t show that price-based competition improves outcomes (if anything, it makes things worse); and the evidence also doesn’t show that GP consortiums improve outcomes (unless you cherry-pick only the positive findings). It’s OK if your reforms aren’t supported by existing evidence: you just shouldn’t claim that they are.
Now Lansley’s Junior Minister Paul Burstow MP has kindly responded, repeating the exact same mistakes again, only more clumsily. From a Minister, this is frightening, so we should see how he does it.
Burstow’s letter:
In Ben Goldacre’s pursuit of the evidence for NHS modernisation (‘Evidence? What evidence?’, 5 February), he appears to have overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans. As he will see, studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients. The evidence on competition demonstrates that when it is well-designed and conducted on the basis of quality (as we are proposing), rather than price, it can drive up quality and efficiency.
We have not sought to understate the achievements of the NHS – but a 2008 study by Martin McKee and Ellen Nolte, citing OECD data, concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations. If the NHS was to perform as well as the best-performing countries, thousands of lives could be saved each year. We make no apology for that.
Finally, Goldacre appears to have misunderstood the aims of our plans. We are not advocating reform for the sake of ideology. The changes we are proposing are designed to put patients first, improve health outcomes, empower clinicians so they can design services that meet the needs of their local communities, and put the NHS on a more sustainable footing so it is better able to meet the challenges of the 21st century.
Paul Burstow MP
Minister for Care Services
The government initially claimed that UK heart attack death rates were twice as bad as those in France. This was an overstatement: they are, but following recent interventions the gap is closing so rapidly that on current trends it will have disappeared entirely by 2012. In response, Burstow cites a 2008 paper by McKee and Nolte which he says ‘concluded that the UK had one of the worst rates of mortality amenable to healthcare among rich nations’.
Burstow either misunderstands or misrepresents this very simple and brief paper. It is a study explicitly looking at time trends, not static figures, and it once again finds that comparing 2003 with 1998, the UK still had fairly high rates of avoidable mortality, but these were falling faster than in all but one of the other eighteen industrialised countries they examined. (Meanwhile, in the US, avoidable mortality improved at a disastrously slow pace, although they spent more money).
This is a paper showing the recent success of the NHS, and the fact that we are discussing such a massive improvement in avoidable mortality from Labour’s first term in government is not my being partisan: this is the paper that was cited by the Tory Minister as evidence, bizarrely, of the NHS’s recent failures.
Next, Burstow says I ‘overlooked the impact assessment we published alongside the health and social care bill, where we present a thorough analysis of the evidence for and against our plans … studies show that GP fundholding and practice-based commissioning delivered shorter waits and fewer referrals to hospitals for patients’.
In its section on GP fundholding, this ‘thorough analysis’ ignores the four peer-reviewed academic papers I described last week, which sadly found no evidence of an overall benefit from GP fundholding. It makes a series of five assertions about outcomes, though not one of these is referenced to any single paper or study, anywhere at all.
I contacted the Department of Health, which ferreted out the sources especially for me. It turned out there was just one, a document from the King’s Fund. It’s not a peer-reviewed academic journal article, but the King’s Fund is pretty good, in my view. If you read this document, once again, as with other reviews of the literature, it finds that the results of GP fundholding were mixed: some things got better, some things got worse.
So, the Minister has cherry-picked only the good findings, from only one report, while ignoring the peer-reviewed literature. Most crucially, he cherry-picks findings he likes while explicitly claiming that he is fairly citing the totality of the evidence from a thorough analysis. By using this approach, I can produce good evidence that I have a magical two-headed coin, if I just disregard all the throws where it comes out tails.
Here is what politicians apparently cannot understand, repeatedly, over and again: it’s fine to make policy based on ideology, whim, faith, principles, and all the other things we’re used to. It’s also fine for evidence to be mixed. And it’s absolutely fine if your reforms aren’t supported by existing evidence: just don’t pretend that they are.
Politicians Can Divine Which Policy Works Best by Using Their Special Magic Politician Beam
Guardian, 22 May 2010
We have accidentally elected a coalition. This week all good citizens are poring over the ‘Programme For Government’, and there is much to be pleased with. Labour wasn’t all about unbridled credit and fun public sector spending sprees: they kept all our emails, kept records of the websites we visited, used ‘anti-terrorism’ legislation on people who plainly weren’t terrorists, and so on.
But most interesting are the noises now being made by the coalition on crime and evidence. ‘We will conduct a full review of sentencing policy,’ they say, ‘to ensure that it is effective in deterring crime, protecting the public, punishing offenders and cutting reoffending. In particular, we will ensure that sentencing for drug use helps offenders come off drugs.’
These are grand promises. Compulsory addiction rehabilitation with ‘Drug Testing and Treatment Orders’ was introduced precisely ten years ago – as an alternative to custodial sentences or simple probation – for those who have committed drug-related crimes. Their implementation without adequate analysis is a graphic example of our failure to run simple trials of social policy.
Any judge making a decision on a criminal’s sentence is in the exact same position as a doctor making a decision on a patient’s treatment: both are choosing an intervention for an individual in front of them, with the intention of producing a particular set of positive outcomes (reduced crime, say, and reduced drug use); both get through a large number of individuals in a month; and in many important situations, neither yet knows which of the available interventions works best.
If you randomly assign a fairly large number of criminals, or patients, to one of two interventions, in situations where you don’t know which intervention would be best, and measure how well they’re doing a year or so later, you instantly discover which intervention is best. Add in the cost, and you know which is most cost-effective. The basic pri
nciples behind this idea are not new, and were first described in the Old Testament, Daniel 1:12.
Before being rolled out nationally in October 2000, DTTOs were extensively piloted in three cities by the Criminal Policy Research Unit of South Bank University, at considerable cost. What insights did this generate? There was no randomisation, and no ‘control’ group of identical criminals given traditional sentences for comparison. Because of that, the only new knowledge generated by these pilots was the revelation that it is possible to set up a DTTO service and run it in some buildings in some cities.
As it happens, when they did follow up the people who had passed through the service, they hadn’t done particularly well. But this finding wasn’t published until after the service had been rolled out. In any case, because there was no randomised comparison group, we have no idea how these participants would have turned out if they’d been given a traditional custodial sentence anyway, so there’s no sense in giving those results even a moment’s thought.
This is a tragedy, and not just because drug use is estimated – with the usual caveats about estimating more nebulous stuff – to be behind 85 per cent of shoplifting, 80 per cent of domestic burglaries, over half of all robberies, and so on. It is also a tragedy because it speaks to motives that will never go away.
We would need a very brave modern politician to say: ‘Look, I want to introduce a new policy, but I honestly don’t know if it will work’. We might need to be forgiving, ourselves, of someone who said this, and actively encourage them to try out their ideas on half of a group of people. We would need a political class that could react to deferred outcomes, with the results dripping out from new interventions over the course of years, perhaps long after the one initiating politician has moved on. Doing all this would revolutionise social policy, in far wider domains than just criminal sentencing. But for now, politicians – and we must share the blame – get further when they use rhetoric, and absolutes.