Hello World

by Hannah Fry

  If the system had been acting entirely autonomously, without a human like Petrov to act as the final arbiter, history would undoubtedly have played out rather differently. Russia would almost certainly have launched what it believed to be retaliatory action and triggered a full-blown nuclear war in the process. If there’s anything we can learn from this story, it’s that the human element does seem to be a critical part of the process: that having a person with the power of veto in a position to review the suggestions of an algorithm before a decision is made is the only sensible way to avoid mistakes.

  After all, only humans will feel the weight of responsibility for their decisions. An algorithm tasked with communicating up to the Kremlin wouldn’t have thought for a second about the potential ramifications of such a decision. But Petrov, on the other hand? ‘I knew perfectly well that nobody would be able to correct my mistake if I had made one.’36

  The only problem with this conclusion is that humans aren’t always that reliable either. Sometimes, like Petrov, they’ll be right to over-rule an algorithm. But often our instincts are best ignored.

  To give you another example from the world of safety, where stories of humans incorrectly over-ruling an algorithm are mercifully rare, that is none the less precisely what happened during an infamous crash on the Smiler rollercoaster at Alton Towers, the UK’s biggest theme park.37

  Back in June 2015, two engineers were called to attend a fault on a rollercoaster. After fixing the issue, they sent an empty carriage around to test everything was working – but failed to notice it never made it back. For whatever reason, the spare carriage rolled backwards down an incline and came to a halt in the middle of the track.

  Meanwhile, unbeknown to the engineers, the ride staff added an extra carriage to deal with the lengthening queues. Once they got the all-clear from the control room, they started loading up the carriages with cheerful passengers, strapping them in and sending the first car off around the track, completely unaware of the empty, stranded carriage sent out by the engineers sitting directly in its path.

  Luckily, the rollercoaster designers had planned for a situation like this, and their safety algorithms worked exactly as planned. To avoid a certain collision, the packed train was brought to a halt at the top of the first climb, setting off an alarm in the control room. But the engineers – confident that they’d just fixed the ride – concluded the automatic warning system was at fault.

  Over-ruling the algorithm wasn’t easy: they both had to agree and simultaneously press a button to restart the rollercoaster. Doing so sent the train full of people over the drop to crash straight into the stranded extra carriage. The result was horrendous. Several people suffered devastating injuries and two teenage girls lost their legs.

  Both of these life-or-death scenarios, Alton Towers and Petrov’s alarm, serve as dramatic illustrations of a much deeper dilemma. In the balance of power between human and algorithm, who – or what – should have the final say?

  Power struggle

  This is a debate with a long history. In 1954, Paul Meehl, a professor of clinical psychology at the University of Minnesota, annoyed an entire generation of humans when he published Clinical versus Statistical Prediction, coming down firmly on one side of the argument.38

  In his book, Meehl systematically compared the performance of humans and algorithms on a whole variety of subjects – predicting everything from students’ grades to patients’ mental health outcomes – and concluded that mathematical algorithms, no matter how simple, will almost always make better predictions than people.

  Countless other studies in the half-century since have confirmed Meehl’s findings. If your task involves any kind of calculation, put your money on the algorithm every time: in making medical diagnoses or sales forecasts, predicting suicide attempts or career satisfaction, and assessing everything from fitness for military service to projected academic performance.39 The machine won’t be perfect, but giving a human a veto over the algorithm would just add more error.fn3

  Perhaps this shouldn’t come as a surprise. We’re not built to compute. We don’t go to the supermarket to find a row of cashiers eyeballing our shopping to gauge how much it should cost. We get an (incredibly simple) algorithm to calculate it for us instead. And most of the time, we’d be better off leaving the machine to it. It’s like the saying among airline pilots that the best flying team has three components: a pilot, a computer and a dog. The computer is there to fly the plane, the pilot is there to feed the dog. And the dog is there to bite the human if it tries to touch the computer.

  But there’s a paradox in our relationship with machines. While we have a tendency to over-trust anything we don’t understand, as soon as we know an algorithm can make mistakes, we also have a rather annoying habit of over-reacting and dismissing it completely, reverting instead to our own flawed judgement. It’s known to researchers as algorithm aversion. People are less tolerant of an algorithm’s mistakes than of their own – even if their own mistakes are bigger.

  It’s a phenomenon that has been demonstrated time and time again in experiments,40 and to some extent, you might recognize it in yourself. Whenever Citymapper says my journey will take longer than I expect it to, I always think I know better (even if most of the time it means I end up arriving late). We’ve all called Siri an idiot at least once, somehow in the process forgetting the staggering technological accomplishment it has taken to build a talking assistant you can hold in your hand. And in the early days of using the mobile GPS app Waze I’d find myself sitting in a traffic jam, having been convinced that taking the back roads would be faster than the route shown. (It almost always wasn’t.) Now I’ve come to trust it and – like Robert Jones and his BMW – I’ll blindly follow it wherever it leads me (although I still think I’d draw the line at going over a cliff).

  This tendency of ours to view things in black and white – seeing algorithms as either omnipotent masters or a useless pile of junk – presents quite a problem in our high-tech age. If we’re going to get the most out of technology, we’re going to need to work out a way to be a bit more objective. We need to learn from Kasparov’s mistake and acknowledge our own flaws, question our gut reactions and be a bit more aware of our feelings towards the algorithms around us. On the flip side, we should take algorithms off their pedestal, examine them a bit more carefully and ask if they’re really capable of doing what they claim. That’s the only way to decide if they deserve the power they’ve been given.

  Unfortunately, all this is often much easier said than done. Oftentimes, we’ll have little say over the power and reach of the algorithms that surround us, even when it comes to those that affect us directly.

  This is particularly true for the algorithms that trade in the most fundamental modern commodity: data. The algorithms that silently follow us around the internet, the ones that are harvesting our personal information, invading our privacy and inferring our character with free rein to subtly influence our behaviour. In that perfect storm of misplaced trust and power and influence, the consequences have the potential to fundamentally alter our society.

  Data

  BACK IN 2004, soon after college student Mark Zuckerberg created Facebook, he had an instant messenger exchange with a friend:

  ZUCK: Yeah so if you ever need info about anyone at Harvard

  ZUCK: Just ask.

  ZUCK: I have over 4,000 emails, pictures, addresses …

  [REDACTED FRIEND’S NAME]: What? How’d you manage that one?

  ZUCK: People just submitted it.

  ZUCK: I don’t know why.

  ZUCK: They ‘trust me’

  ZUCK: Dumb fucks1

  In the wake of the 2018 Facebook scandal, these words were repeatedly reprinted by journalists wanting to hint at a Machiavellian attitude to privacy within the company. Personally, I think we can be a little more generous when interpreting the boastful comments of a 19-year-old. But I also think that Zuckerberg is wrong. People weren’t just giving him th
eir details. They were submitting them as part of an exchange. In return, they were given access to an algorithm that would let them freely connect with friends and family, a space to share their lives with others. Their own private network in the vastness of the World Wide Web. I don’t know about you, but at the time I certainly thought that was a fair swap.

  There’s just one issue with that logic: we’re not always aware of the longer-term implications of that trade. It’s rarely obvious what our data can do, or, when fed into a clever algorithm, just how valuable it can be. Nor, in turn, how cheaply we were bought.

  Every little helps

  Supermarkets were among the first to recognize the value of an individual’s data. In a sector where companies are continually fighting for the customer’s attention – for tiny margins of preference that will nudge people’s buying behaviour into loyalty to their brand – every slim improvement can add up to an enormous advantage. This was the motivation behind a ground-breaking trial run in 1993 by the British supermarket Tesco.

  Under the guidance of husband-and-wife team Edwina Dunn and Clive Humby, and beginning in certain selected stores, Tesco released its brand-new Clubcard – a plastic card, the size and shape of a credit card, that customers could present at a checkout when paying for their shopping. The exchange was simple. For each transaction using a Clubcard, the customer would collect points that they could use against future purchases in store, while Tesco would take a record of the sale and associate it with the customer’s name.2

  The data gathered in that first Clubcard trial was extremely limited. Along with the customer’s name and address, the scheme only recorded what they spent and when, not which items were in their basket. None the less, from this modest harvest of data Dunn and Humby reaped some phenomenally valuable insights.

  They discovered that a small handful of loyal customers accounted for a massive amount of their sales. They saw, postcode by postcode, how far people were willing to travel to their stores. They uncovered the neighbourhoods where the competition was winning and neighbourhoods where Tesco had the upper hand. The data revealed which customers came back day after day, and which saved their shopping for weekends. Armed with that knowledge, they could get to work nudging their customers’ buying behaviour, by sending out a series of coupons to the Clubcard users in the post. High spenders were given vouchers ranging from £3 to £30. Low spenders were sent a smaller incentive of £1 to £10. And the results were staggering. Nearly 70 per cent of the coupons were redeemed, and while in the stores, customers filled up their baskets: people who had Clubcards spent 4 per cent more overall than those who didn’t.

  On 22 November 1994, Clive Humby presented the findings from the trial to the Tesco board. He showed them the data, the response rates, the evidence of customer satisfaction, the sales boosts. The board listened in silence. At the end of the presentation, the chair was the first person to speak. ‘What scares me about this,’ he said, ‘is that you know more about my customers in three months than I know in 30 years.’3

  Clubcard was rolled out to all customers of Tesco and is widely credited with putting the company ahead of its main rival Sainsbury’s, to become the biggest supermarket in the UK. As time wore on, the data collected became more detailed, making customers’ buying habits easier to target.

  Early in the days of online shopping, the team introduced a feature known as ‘My Favourites’, in which any items that were bought while using the loyalty card would appear prominently when the customer logged on to the Tesco website. Like the Clubcard itself, the feature was a roaring success. People could quickly find the products they wanted without having to navigate through the various pages. Sales went up, customers were happy.

  But not all of them. Shortly after the launch of the feature, one woman contacted Tesco to complain that her data was wrong. She’d been shopping online and seen condoms among her list of ‘My Favourites’. They couldn’t be her husband’s, she explained, because he didn’t use them. At her request, the Tesco analysts looked into the data and discovered that her list was accurate. However, rather than be the cause of a marital rift, they took the diplomatic decision to apologize for ‘corrupted data’ and remove the offending items from her favourites.

  According to Clive Humby’s book on Tesco, this has now become an informal policy within the company. Whenever something comes up that is just a bit too revealing, they apologize and delete the data. It’s a stance that’s echoed by Eric Schmidt, who, while serving as the executive chairman of Google, said he tries to think of things in terms of an imaginary creepy line. ‘The Google policy is to get right up to the creepy line but not cross it.’4

  But collect enough data and it’s hard to know what you’ll uncover. Groceries aren’t just what you consume. They’re personal. Look carefully enough at someone’s shopping habits and they’ll often reveal all kinds of detail about who they are as a person. Sometimes – as in the case of the condoms – it’ll be things you’d rather not know. But more often than not, lurking deep within the data, those slivers of hidden insight can be used to a company’s advantage.

  Target market

  Back in 2002, the American discount superstore Target started looking for unusual patterns in its data.5 Target sells everything from milk and bananas to cuddly toys and garden furniture, and – like pretty much every other retailer since the turn of the millennium – has ways of using credit card numbers and survey responses to tie customers to everything they’ve ever bought in the store, enabling them to analyse what people are buying.

  In a story that – as US readers won’t need telling – became infamous across the country, Target realized that a spike in a female customer’s purchases of unscented body lotion would often precede her signing up to the in-store baby-shower registry. It had found a signal in the data. As women entered their second trimester and started to worry about stretch marks, their buying of moisturizer to keep their skin supple left a hint of what was to come. Scroll backwards further in time, and these same women would be popping into Target to stock up on various vitamins and supplements, like calcium and zinc. Scroll forwards in time and the data would even suggest when the baby was due – marked by the woman buying extra-big bags of cotton wool from the store.6

  Expectant mothers are a retailer’s dream. Lock in her loyalty while she’s pregnant and there’s a good chance she’ll continue to use your products long after the birth of her child. After all, shopping habits are quick to form when a hungry screaming baby is demanding your attention during your weekly shop. Insights like this could be hugely valuable in giving Target a head start over other brands in attracting her business.

  From there it was simple. Target ran an algorithm that would score its female customers on the likelihood they were pregnant. If that probability tipped past a certain threshold, the retailer would automatically send out a series of coupons to the woman in question, full of things she might find useful: nappies, lotions, baby wipes and so on.

  So far, so uncontroversial. But then, around a year after the tool was first introduced, a father of a teenage girl stormed into a Target store in Minneapolis demanding to see the manager. His daughter had been sent some pregnancy coupons in the post and he was outraged that the retailer seemed to be normalizing teenage pregnancy. The manager of the store apologized profusely and called the man’s home a few days later to reiterate the company’s regret about the whole affair. But by then, according to a story in the New York Times, the father had an apology of his own to make.

  ‘I had a talk with my daughter,’ he told the manager. ‘It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August.’

  I don’t know about you, but for me, an algorithm that will inform a parent that their daughter is pregnant before they’ve had a chance to learn about it in person takes a big step across the creepy line. But this embarrassment wasn’t enough to persuade Target to scrap the tool altogether.

  A Tar
get executive explained: ‘We found out that as long as a pregnant woman thinks she hasn’t been spied on, she’ll use the coupons. She just assumes that everyone else on her block got the same mailer for diapers and cribs. As long as we don’t spook her, it works.’

  So, Target still has a pregnancy predictor running behind the scenes – as most retailers do now. The only difference is that it will mix in the pregnancy-related coupons with other more generic items so that the customers don’t notice they’ve been targeted. An advertisement for a crib might appear opposite some wine glasses. Or a coupon for baby clothes will run alongside an ad for some cologne.

  Target is not alone in using these methods. Stories of what can be inferred from your data rarely hit the press, but the algorithms are out there, quietly hiding behind the corporate front lines. About a year ago, I got chatting to a chief data officer of a company that sells insurance. They had access to the full detail of people’s shopping habits via a supermarket loyalty scheme. In their analysis, they’d discovered that home cooks were less likely to claim on their home insurance, and were therefore more profitable. It’s a finding that makes good intuitive sense. There probably isn’t much crossover between the group of people who are willing to invest time, effort and money in creating an elaborate dish from scratch and the group who would let their children play football in the house. But how did they know which shoppers were home cooks? Well, there were a few items in someone’s basket that were linked to low claim rates. The most significant, he told me, the one that gives you away as a responsible, houseproud person more than any other, was fresh fennel.

  If that’s what you can infer from people’s shopping habits in the physical world, just imagine what you might be able to infer if you had access to more data. Imagine how much you could learn about someone if you had a record of everything they did online.