Weapons of Math Destruction

by Cathy O'Neil

  ZestFinance buys data that shows whether applicants have kept up with their cell phone bills, along with plenty of other publicly available or purchased data. As Merrill promised, the company’s rates are lower than those charged by most payday lenders. A typical $500 loan at ZestFinance costs $900 after twenty-two weeks—60 percent lower than the industry standard.

  It’s an improvement, but is it fair? The company’s algorithms process up to ten thousand data points per applicant, including unusual observations, such as whether applicants use proper spelling and capitalization on their application form, how long it takes them to read it, and whether they bother to look at the terms and conditions. “Rule followers,” the company argues, are better credit risks.

  That may be true. But punctuation and spelling mistakes also point to low education, which is highly correlated with class and race. So when poor people and immigrants qualify for a loan, their substandard language skills might drive up their fees. If they then have trouble paying those fees, this might validate that they were a high risk to begin with and might further lower their credit scores. It’s a vicious feedback loop, and paying bills on time plays only a bit part.

  When new ventures are built on WMDs, troubles are bound to follow, even when the players have the best intentions. Take the case of the “peer-to-peer” lending industry. It started out in the last decade with the vision of borrowers and lenders finding each other on matchmaking platforms. This would represent the democratization of banking. More people would get loans, and at the same time millions of everyday people would become small-time bankers and make a nice return. Both sides would bypass the big greedy banks.

  One of the first peer-to-peer exchanges, Lending Club, launched as an application on Facebook in 2006 and received funding a year later to become a new type of bank. To calculate the borrower’s risk, Lending Club blended the traditional credit report with data gathered from around the web. Their algorithm, in a word, generated e-scores, which they claimed were more accurate than credit scores.

  Lending Club and its chief rival, Prosper, are still tiny. They’ve generated less than $10 billion in loans, which is but a speck in the $3 trillion consumer lending market. Yet they’re attracting loads of attention. Executives from Citigroup and Morgan Stanley serve as directors of peer-to-peer players, and Wells Fargo’s investment fund is the largest investor in Lending Club. Lending Club’s stock offering in December of 2014 was the biggest tech IPO of the year. It raised $870 million and reached a valuation of $9 billion, making it the fifteenth most valuable bank in America.

  The fuss has little to do with democratizing capital or cutting out the middleman. According to a report in Forbes, institutional money now accounts for more than 80 percent of all the activity on peer-to-peer platforms. For big banks, the new platforms provide a convenient alternative to the tightly regulated banking economy. Working through peer-to-peer systems, a lender can analyze nearly any data it chooses and develop its own e-scores. It can develop risk correlations for neighborhoods, zip codes, and the stores customers shop at—all without having to send them embarrassing letters explaining why.

  And what does that mean for us? With the relentless growth of e-scores, we’re batched and bucketed according to secret formulas, some of them fed by portfolios loaded with errors. We’re viewed not as individuals but as members of tribes, and we’re stuck with that designation. As e-scores pollute the sphere of finance, opportunities dim for the have-nots. In fact, compared to the slew of WMDs running amok, the prejudiced loan officer of yesteryear doesn’t look all that bad. At the very least, a borrower could attempt to read his eyes and appeal to his humanity.

  * * *

  * Even so, I should add, fixing them can be a nightmare. A Mississippi resident named Patricia Armour tried for two years to get Experian to expunge from her file a $40,000 debt she no longer owed. It took a call to Mississippi’s attorney general, she told the New York Times, before Experian corrected her record.

  Late in the nineteenth century, a renowned statistician named Frederick Hoffman created a potent WMD. It’s very likely that Hoffman, a German who worked for the Prudential Life Insurance Company, meant no harm. Later in his life, his research contributed mightily to public health. He did valuable work on malaria and was among the first to associate cancer with tobacco. Yet on a spring day in 1896, Hoffman published a 330-page report that set back the cause of racial equality in the United States and reinforced the status of millions as second-class citizens. His report used exhaustive statistics to make the case that the lives of black Americans were so precarious that the entire race was uninsurable.

  Hoffman’s analysis, like many of the WMDs we’ve been discussing, was statistically flawed. He confused causation with correlation, so that the voluminous data he gathered served only to confirm his thesis: that race was a powerful predictor of life expectancy. Racism was so ingrained in his thinking that he apparently never stopped to consider whether poverty and injustice might have something to do with the death rate of African Americans, whether the lack of decent schools, modern plumbing, safe workplaces, and access to health care might kill them at a younger age.

  Hoffman also made a fundamental statistical error. Like the presidential commission that issued the 1983 Nation at Risk report, Hoffman neglected to stratify his results. He saw blacks only as a large and homogeneous group. So he failed to separate them into different geographical, social, or economic cohorts. For him, a black schoolteacher leading an orderly life in Boston or New York was indistinguishable from a sharecropper laboring twelve hours a day barefoot in the Mississippi Delta. Hoffman was blinded by race.

  And so was his industry. With time, of course, insurers advanced a bit in their thinking and sold policies to African American families. After all, there was money to be made. But they clung for decades to Hoffman’s idea that entire groups of people were riskier than others—and some of them too risky. Insurance companies as well as bankers delineated neighborhoods where they would not invest. This cruel practice, known as redlining, has been outlawed by various pieces of legislation, including the Fair Housing Act of 1968.

  Nearly a half century later, however, redlining is still with us, though in far more subtle forms. It’s coded into the latest generation of WMDs. Like Hoffman, the creators of these new models confuse correlation with causation. They punish the poor, and especially racial and ethnic minorities. And they back up their analysis with reams of statistics, which give them the studied air of evenhanded science.

  On this algorithmic voyage through life, we’ve clawed our way through education and we’ve landed a job (even if it is one that runs us on a chaotic schedule). We’ve taken out loans and seen how our creditworthiness is a stand-in for other virtues or vices. Now it’s time to protect our most treasured assets—our home and car and our family’s health—and make arrangements for those we one day leave behind.

  Insurance grew out of actuarial science, a discipline whose roots reach back to the seventeenth century. This was a period in which Europe’s growing bourgeoisie was acquiring great wealth. It allowed many the luxury, for the first time, to think ahead to future generations.

  While advances in math were providing the tools necessary to make predictions, an early generation of data hounds was looking for new things to count. One was a draper in London named John Graunt. He went through birth and death records and in 1682 came up with the first study of the mortality rates of an entire community of people. He calculated, for example, that children in London faced a 6 percent death risk in each of the first six years of their lives. (And with statistics, he was able to dispel the myth that the plague swept through every year a new monarch came into power.) For the first time, mathematicians could calculate the most probable arc of a person’s life. These numbers didn’t work for individuals, of course. But with big enough numbers, the average and range were predictable.

  Mathematicians didn’t pretend to foresee the fate of each individual. That was unkn
owable. But they could predict the prevalence of accidents, fires, and deaths within large groups of people. Over the following three centuries, a vast insurance industry grew around these predictions. The new industry gave people, for the first time, the chance to pool their collective risk, protecting individuals when misfortune struck.

  Now, with the evolution of data science and networked computers, insurance is facing fundamental change. With ever more information available—including the data from our genomes, the patterns of our sleep, exercise, and diet, and the proficiency of our driving—insurers will increasingly calculate risk for the individual and free themselves from the generalities of the larger pool. For many, this is a welcome change. A health enthusiast today can demonstrate, with data, that she sleeps eight hours a night, walks ten miles a day, and eats little but green vegetables, nuts, and fish oil. Why shouldn’t she get a break on her health insurance?

  The move toward the individual, as we’ll see, is embryonic. But already insurers are using data to divide us into smaller tribes, to offer us different products and services at varying prices. Some might call this customized service. The trouble is, it’s not individual. The models place us into groups we cannot see, whose behavior appears to resemble ours. Regardless of the quality of the analysis, its opacity can lead to gouging.

  Take auto insurance. In 2015, researchers at Consumer Reports conducted an extensive nationwide study looking for disparities in pricing. They analyzed more than two billion price quotes from all the major insurers for hypothetical customers from every one of the 33,419 zip codes in the country. What they found was wildly unfair, and rooted—as we saw in the last chapter—in credit scores.

  Insurers draw these scores from credit reports, and then, using the insurer’s proprietary algorithm, create their own ratings, or e-scores. These are proxies for responsible driving. But Consumer Reports found that the e-scores, which include all sorts of demographic data, often count for more than the driver’s record. In other words, how you manage money can matter more than how you drive a car. In New York State, for example, a dip in a driver’s credit rating from “excellent” to merely “good” could jack up the annual cost of insurance by $255. And in Florida, adults with clean driving records and poor credit scores paid an average of $1,552 more than the same drivers with excellent credit and a drunk driving conviction.

  We’ve already discussed how the growing reliance on credit scores across the economy works against the poor. This is yet another example of that trend, and an egregious one—especially since auto insurance is mandatory for anyone who drives. What’s different here is the focus on the proxy when far more relevant data is available. I cannot imagine a more meaningful piece of data for auto insurers than a drunk driving record. It is evidence of risk in precisely the domain they’re attempting to predict. It’s far better than other proxies they consider, such as a high school student’s grade point average. Yet it can count far less in their formula than a score drawn from financial data thrown together on a credit report (which, as we’ve seen, is sometimes erroneous).

  So why would their models zero in on credit scores? Well, like other WMDs, automatic systems can plow through credit scores with great efficiency and at enormous scale. But I would argue that the chief reason has to do with profits. If an insurer has a system that can pull in an extra $1,552 a year from a driver with a clean record, why change it? The victims of their WMD, as we’ve seen elsewhere, are more likely to be poor and less educated, a good number of them immigrants. They’re less likely to know that they’re being ripped off. And in neighborhoods with more payday loan offices than insurance brokers, it’s harder to shop for lower rates. In short, while an e-score might not correlate with safe driving, it does create a lucrative pool of vulnerable drivers. Many of them are desperate to drive—their jobs depend on it. Overcharging them is good for the bottom line.

  From the auto insurer’s perspective, it’s a win-win. A good driver with a bad credit score is low risk and superhigh reward. What’s more, the company can use some of the proceeds from that policy to address the inefficiencies in the model. Those might include the drivers with pristine credit reports who pay low premiums and crash their cars while drunk.

  That may sound a tad cynical. But consider the price optimization algorithm at Allstate, the insurer self-branded as “the Good Hands People.” According to a watchdog group, the Consumer Federation of America, Allstate analyzes consumer and demographic data to determine the likelihood that customers will shop for lower prices. If they aren’t likely to, it makes sense to charge them more. And that’s just what Allstate does.

  It gets worse. In a filing to the Wisconsin Department of Insurance, the CFA listed one hundred thousand microsegments in Allstate’s pricing schemes. These pricing tiers are based on how much each group can be expected to pay. Consequently, some receive discounts of up to 90 percent off the average rate, while others face an increase of 800 percent. “Allstate’s insurance pricing has become untethered from the rules of risk-based premiums and from the rule of law,” said J. Robert Hunter, CFA’s director of insurance and the former Texas insurance commissioner. Allstate responded that the CFA’s charges were inaccurate. The company did concede, however, that “marketplace considerations, consistent with industry practices, have been appropriate in developing insurance prices.” In other words, its models study a host of proxies to calculate how much to charge customers. And the rest of the industry does, too.

  The resulting pricing is unfair. This abuse could not occur if insurance pricing were transparent and customers could easily comparison-shop. But like other WMDs, it is opaque. Every person gets a different experience, and the models are optimized to draw as much money as they can from the desperate and the ignorant. The result—another feedback loop—is that poor drivers who can least afford outrageous premiums are squeezed for every penny they have. The model is fine-tuned to draw as much money as possible from this subgroup. Some of them, inevitably, fall too far, defaulting on their auto loans, credit cards, or rent. That further punishes their credit scores, which no doubt drops them into an even more forlorn microsegment.

  When Consumer Reports issued its damning report on the auto insurers, it also launched a campaign directed at the National Association of Insurance Commissioners (NAIC), complete with its own Twitter campaign: @NAIC_News to Insurance Commissioners: Price me by how I drive, not by who you think I am! #FixCarInsurance.

  The underlying idea was that drivers should be judged by their records—their number of speeding tickets, or whether they’ve been in an accident—and not by their consumer patterns or those of their friends or neighbors. Yet in the age of Big Data, urging insurers to judge us by how we drive means something entirely new.

  Insurance companies now have manifold ways to study drivers’ behavior in exquisite detail. For a preview, look no further than the trucking industry.

  These days, many trucks carry an electronic logging device that registers every turn, every acceleration, every time they touch the brakes. And in 2015, Swift Transportation, the nation’s largest trucking company, started to install cameras pointed in two directions, one toward the road ahead, the other at the driver’s face.

  The stated goal of this surveillance is to reduce accidents. About seven hundred truckers die on American roads every year. And their crashes also claim the lives of many in other vehicles. In addition to the personal tragedy, this costs lots of money. The average cost of a fatal crash, according to the Federal Motor Carrier Safety Administration, is $3.5 million.

  But with such an immense laboratory for analytics at their fingertips, trucking companies aren’t stopping at safety. If you combine geoposition, onboard tracking technology, and cameras, truck drivers deliver a rich and constant stream of behavioral data. Trucking companies can now analyze different routes, assess fuel management, and compare results at different times of the day and night. They can even calculate ideal speeds for different road surfaces. And they use this
data to figure out which patterns provide the most revenue at the lowest cost.

  They can also compare individual drivers. Analytics dashboards give each driver a scorecard. With a click or two, a manager can identify the best and worst performers across a broad range of metrics. Naturally, this surveillance data can also calculate the risk for each driver.

  This promise is not lost on the insurance industry. Leading insurers including Progressive, State Farm, and Travelers are already offering drivers a discount on their rates if they agree to share their driving data. A small telemetric unit in the car, a simple version of the black boxes in airplanes, logs the speed of the car and how the driver brakes and accelerates. A GPS monitor tracks the car’s movements.

  In theory, this meets the ideal of the Consumer Reports campaign. The individual driver comes into focus. Consider eighteen-year-olds. Traditionally they pay sky-high rates because their age group, statistically, indulges in more than its share of recklessness. But now, a high school senior who avoids jackrabbit starts, drives at a consistent pace under the speed limit, and eases to a stop at red lights might get a discounted rate. Insurance companies have long given an edge to young motorists who finish driver’s ed or make the honor roll. Those are proxies for responsible driving. But driving data is the real thing. That’s better, right?

  There are a couple of problems. First, if the system attributes risk to geography, poor drivers lose out. They are more likely to drive in what insurers deem risky neighborhoods. Many also have long and irregular commutes, which translates into higher risk.

  Fine, you might say. If poor neighborhoods are riskier, especially for auto theft, why should insurance companies ignore that information? And if longer commutes increase the chance of accidents, that’s something the insurers are entitled to consider. The judgment is still based on the driver’s behavior, not on extraneous details like her credit rating or the driving records of people her age. Many would consider that an improvement.