Weapons of Math Destruction

by Cathy O'Neil

  Once Ghani’s data team understood this small group of voters, their desires, their fears, and what it took to change their behavior, the next challenge was to find millions of other voters (and donors) who resembled them. This involved plowing through the consumer data and demographics of the voters they had interviewed and building mathematical profiles of them. Then it was just a matter of scouring national databases, finding people with similar profiles, and placing them into the same buckets.

  The campaign could then target each group with advertisements, perhaps on Facebook or the media sites they visited, to see if they responded as expected. They carried out the same kind of A/B testing that Google uses to see which shade of blue garners more clicks on a button. Trying different approaches, they found, for example, that e-mail subject lines reading only “Hey!” bugged people but also led to more engagement and sometimes more donations. Through thousands of tests and tweaks, the campaign finally sized up its audience—including an all-important contingent of fifteen million swing voters.

  Throughout this process, each campaign developed profiles of American voters. Each profile contained numerous scores, which not only gauged their value as a potential voter, volunteer, and donor but also reflected their stances on different issues. One voter might have a high score on environmental issues but a low one on national security or international trade. These political profiles are very similar to those that Internet companies, like Amazon and Netflix, use to manage their tens of millions of customers. Those companies’ analytics engines churn out nearly constant cost/benefit analyses to maximize their revenue per customer.

  Four years later, Hillary Clinton’s campaign built upon the methodology established by Obama’s team. It contracted a microtargeting start-up, the Groundwork, financed by Google chairman Eric Schmidt and run by Michael Slaby, the chief technology officer of Obama’s 2012 campaign. The goal, according to a report in Quartz, was to build a data system that would create a political version of systems that companies like Salesforce.​com develop to manage their millions of customers.

  The appetite for fresh and relevant data, as you might imagine, is intense. And some of the methods used to gather it are unsavory, not to mention intrusive. In late 2015, the Guardian reported that a political data firm, Cambridge Analytica, had paid academics in the United Kingdom to amass Facebook profiles of US voters, with demographic details and records of each user’s “likes.” They used this information to develop psychographic analyses of more than forty million voters, ranking each on the scale of the “big five” personality traits: openness, conscientiousness, extroversion, agreeableness, and neuroticism. Groups working with the Ted Cruz presidential campaign then used these studies to develop television commercials targeted for different types of voters, placing them in programming they’d be most likely to watch. When the Republican Jewish Coalition was meeting at the Venetian in Las Vegas in May 2015, for instance, the Cruz campaign unleashed a series of web-based advertisements visible only inside the hotel complex that emphasized Cruz’s devotion to Israel and its security.

  I should mention here that not all of these targeting campaigns have proven to be effective. Some, no doubt, are selling little more than snake oil. The microtargeters, after all, are themselves marketing to campaigns and political action groups with millions of dollars to spend. They sell them grand promises of priceless databases and pinpoint targeting, many of which are bound to be exaggerated. So in this sense the politicians not only purvey questionable promises but also consume them (at exorbitant expense). That said, as the Obama team demonstrated, some of these methods are fruitful. And so the industry—serious data scientists and hucksters alike—zeros in on voters.

  Political microtargeters, however, face unique constraints, which make their work far more complex. The value of each voter, for example, rises or falls depending on the probability that his or her state will be in play. A swing voter in a swing state, like Florida, Ohio, or Nevada, is highly valuable. But if polls show the state tilting decisively to either blue or red, that voter’s value plummets, and the marketing budget is quickly shifted toward other voters whose value is climbing.

  In this sense, we can think of the voting public very much as we think of financial markets. With the flow of information, values rise and fall, as do investments. In these new political markets, each one of us represents a stock with its own fluctuating price. And each campaign must decide if and how to invest in us. If we merit the investment, then they decide not only what information to feed us but also how much and how to deliver it.

  Similar calculations, on a macro scale, have been going on for decades, as campaigns plot their TV spending. As polling num bers change, they might cut ads in Pittsburgh and move those dollars to Tampa or Las Vegas. But with microtargeting, the focus shifts from the region to the individual. More important, that individual alone sees the customized version of the politician.

  The campaigns use similar analysis to identify potential donors and to optimize each one. Here it gets complicated, because many of the donors themselves are carrying out their own calculations. They want the biggest bang for their buck. They know that if they immediately hand over the maximum contribution the campaign will view them as “fully tapped” and therefore irrelevant. But refusing to give any money will also render them irrelevant. So many give a drip-feed of money based on whether the messages they hear are ones they agree with. For them, managing a politician is like training a dog with treats. This training effect is all the more powerful for contributors to Super PACS, which do not limit political contributions.

  The campaigns, of course, are well aware of this tactic. With microtargeting, they can send each of those donors the information most likely to pry more dollars from their bank accounts. And these messages will vary from one donor to the next.

  These tactics aren’t limited to campaigns. They infect our civic life, with lobbyists and interest groups now using these targeting methods to carry out their dirty work. In 2015, the Center for Medical Progress, an antiabortion group, posted videos featuring what they claimed was an aborted fetus at a Planned Parenthood clinic. The videos asserted that Planned Parenthood doctors were selling baby parts for research, and they spurred a wave of protest, and a Republican push to eliminate the organization’s funding.

  Research later showed that the video had been doctored: the so-called fetus was actually a photo of a stillborn baby born to a woman in rural Pennsylvania. And Planned Parenthood does not sell fetal tissue. The Center for Medical Progress admitted that the video contained misinformation. That weakened its appeal for a mass market. But with microtargeting, antiabortion activists could continue to build an audience for the video, despite the flawed premise, and use it to raise funds to fight Planned Parenthood.

  While that campaign launched into public view, hundreds of others continue to hover below the surface, addressing individual voters. These quieter campaigns are equally deceptive and even less accountable. And they deliver ideological bombs that politicians will only hint at on the record. According to Zeynep Tufekci, a techno-sociologist and professor at the University of North Carolina, these groups pinpoint vulnerable voters and then target them with fear-mongering campaigns, scaring them about their children’s safety or the rise of illegal immigration. At the same time, they can keep those ads from the eyes of voters likely to be turned off (or even disgusted) by such messaging.

  Successful microtargeting, in part, explains why in 2015 more than 43 percent of Republicans, according to a survey, still believed the lie that President Obama is a Muslim. And 20 percent of Americans believed he was born outside the United States and, consequently, an illegitimate president. (Democrats may well spread their own disinformation in microtargeting, but nothing that has surfaced matches the scale of the anti-Obama campaigns.)

  Even with the growth of microtargeting, political campaigns are still directing 75 percent of their media buy, on average, to television. You might think that this would
have an equalizing effect, and it does. Television delivers the broader, and accountable, messaging, while microtargeting does its work in the shadows. But even television is moving toward personalized advertising. New advertising companies like Simulmedia, in New York, assemble TV viewers into behavioral buckets, so that advertisers can target audiences of like-minded people, whether hunters, pacifists, or buyers of tank-sized SUVs. As television and the rest of the media move toward profiling their viewers, the potential for political microtargeting grows.

  As this happens, it will become harder to access the political messages our neighbors are seeing—and as a result, to understand why they believe what they do, often passionately. Even a nosy journalist will struggle to track down the messaging. It is not enough simply to visit the candidate’s web page, because they, too, automatically profile and target each visitor, weighing everything from their zip codes to the links they click on the page, even the photos they appear to look at. It’s also fruitless to create dozens of “fake” profiles, because the systems associate each real voter with deep accumulated knowledge, including purchasing records, addresses, phone numbers, voting records, and even social security numbers and Facebook profiles. To convince the system it’s real, each fake would have to come with its own load of data. Fabricating one would require far too much work for a research project (and in the worst-case scenario it might get the investigator tangled up in fraud).

  The result of these subterranean campaigns is a dangerous imbalance. The political marketers maintain deep dossiers on us, feed us a trickle of information, and measure how we respond to it. But we’re kept in the dark about what our neighbors are being fed. This resembles a common tactic used by business negotiators. They deal with different parties separately so that none of them knows what the other is hearing. This asymmetry of information prevents the various parties from joining forces—which is precisely the point of a democratic government.

  This growing science of microtargeting, with its profiles and predictions, fits all too neatly into our dark collection of WMDs. It is vast, opaque, and unaccountable. It provides cover to politicians, encouraging them to be many things to many people.

  The scoring of individual voters also undermines democracy, making a minority of voters important and the rest little more than a supporting cast. Indeed, looking at the models used in presidential elections, we seem to inhabit a shrunken country. As I write this, the entire voting population that matters lives in a handful of counties in Florida, Ohio, Nevada, and a few other swing states. Within those counties is a small number of voters whose opinions weigh in the balance. I might point out here that while many of the WMDs we’ve been looking at, from predatory ads to policing models, deliver most of their punishment to the struggling classes, political microtargeting harms voters of every economic class. From Manhattan to San Francisco, rich and poor alike find themselves disenfranchised (though the truly affluent, of course, can more than compensate for this with campaign contributions).

  In any case, the entire political system—the money, the attention, the fawning—turns to targeted voters like a flower following the sun. The rest of us are virtually ignored (except for fund-raising come-ons). The programs have already predicted our voting behavior, and any attempt to change it is not worth the investment.*2

  This creates a nefarious feedback loop. The disregarded voters are more likely to grow disenchanted. The winners know how to play the game. They get the inside story, while the vast majority of consumers receive only market-tested scraps.

  Indeed, there is an added asymmetry. People who are expected to be voters but who, for one reason or another, skip an election find themselves lavished with attention the next time round. They still seem to brim with high voting potential. But those expected not to vote are largely ignored. The systems are searching for the cheapest votes to convert, with the highest return for each dollar spent. And nonvoters often look expensive. This dynamic prods a certain class of people to stay active and lets the rest lie fallow forever.

  As is often the case with WMDs, the very same models that inflict damage could be used to humanity’s benefit. Instead of targeting people in order to manipulate them, it could line them up for help. In a mayoral race, for example, a microtargeting campaign might tag certain voters for angry messages about unaffordable rents. But if the candidate knows these voters are angry about rent, how about using the same technology to identify the ones who will most benefit from affordable housing and then help them find it?

  With political messaging, as with most WMDs, the heart of the problem is almost always the objective. Change that objective from leeching off people to helping them, and a WMD is disarmed—and can even become a force for good.

  * * *

  *1 Similarly, consumer websites are much more likely to offer discounts to people who are not already logged in. This is another reason to clear your cookies regularly.

  *2 At the federal level, this problem could be greatly alleviated by abolishing the Electoral College system. It’s the winner-take-all mathematics from state to state that delivers so much power to a relative handful of voters. It’s as if in politics, as in economics, we have a privileged 1 percent. And the money from the financial 1 percent underwrites the microtargeting to secure the votes of the political 1 percent. Without the Electoral College, by contrast, every vote would be worth exactly the same. That would be a step toward democracy.

  In this march through a virtual lifetime, we’ve visited school and college, the courts and the workplace, even the voting booth. Along the way, we’ve witnessed the destruction caused by WMDs. Promising efficiency and fairness, they distort higher education, drive up debt, spur mass incarceration, pummel the poor at nearly every juncture, and undermine democracy. It might seem like the logical response is to disarm these weapons, one by one.

  The problem is that they’re feeding on each other. Poor people are more likely to have bad credit and live in high-crime neighborhoods, surrounded by other poor people. Once the dark universe of WMDs digests that data, it showers them with predatory ads for subprime loans or for-profit schools. It sends more police to arrest them, and when they’re convicted it sentences them to longer terms. This data feeds into other WMDs, which score the same people as high risks or easy targets and proceed to block them from jobs, while jacking up their rates for mortgages, car loans, and every kind of insurance imaginable. This drives their credit rating down further, creating nothing less than a death spiral of modeling. Being poor in a world of WMDs is getting more and more dangerous and expensive.

  The same WMDs that abuse the poor also place the comfortable classes of society in their own marketing silos. They jet them off to vacations in Aruba and wait-list them at Wharton. For many of them, it can feel as though the world is getting smarter and easier. Models highlight bargains on prosciutto and chianti, recommend a great movie on Amazon Prime, or lead them, turn by turn, to a café in what used to be a “sketchy” neighborhood. The quiet and personal nature of this targeting keeps society’s winners from seeing how the very same models are destroying lives, sometimes just a few blocks away.

  Our national motto, E Pluribus Unum, means “Out of Many, One.” But WMDs reverse the equation. Working in darkness, they carve one into many, while hiding us from the harms they inflict upon our neighbors near and far. And those harms are legion. They unfold when a single mother can’t arrange child care fast enough to adapt to her work schedule, or when a struggling young person is red-lighted for an hourly job by a workplace personality test. We see them when a poor minority teenager gets stopped, roughed up, and put on warning by the local police, or when a gas station attendant who lives in a poor zip code gets hit with a higher insurance bill. It’s a silent war that hits the poor hardest but also hammers the middle class. Its victims, for the most part, lack economic power, access to lawyers, or well-funded political organizations to fight their battles. The result is widespread damage that all too often passes for inevitability.
/>
  We cannot count on the free market itself to right these wrongs. To understand why, let’s compare WMDs to another scourge our society has been grappling with, homophobia.

  In September of 1996, two months before his reelection, President Bill Clinton signed the Defense of Marriage Act. This law, defining marriage as between one man and one woman, promised to firm up support for the president in conservative patches of battleground states, including Ohio and Florida.

  Only a week later, the tech giant IBM announced that it would provide medical benefits to the same-sex partners of its employees. You might wonder why Big Blue, a pillar of the corporate establishment, would open this door and invite controversy when a putatively progressive American president was moving in the opposite direction.

  The answer has to do with the bottom line. In 1996, the Internet gold rush was just taking off, and IBM was battling for brainpower with Oracle, Microsoft, Hewlett-Packard, and a host of start-ups, including Amazon and Yahoo. Most of those other companies were already providing benefits to same-sex partners and attracting gay and lesbian talent. IBM could not afford to miss out. “In terms of business competitiveness, it made sense for us,” an IBM spokesperson told BusinessWeek at the time.

  If we think about human resources policies at IBM and other companies as algorithms, they codified discrimination for decades. The move to equalize benefits nudged them toward fairness. Since then, gays and lesbians have registered impressive progress in many domains. This progress is uneven, of course. Many gay, lesbian, and transgender Americans are still victims of prejudice, violence, and WMDs. This is especially true among poor and minority populations. Still, as I write this, a gay man, Tim Cook, is the chief executive of Apple, the most valuable company on earth. And if he so chooses, he has the constitutional right to marry a man.