Weapons of Math Destruction

by Cathy O'Neil

  Students in the Chinese city of Zhongxiang had a reputation for acing the national standardized test, or gaokao, and winning places in China’s top universities. They did so well, in fact, that authorities began to suspect they were cheating. Suspicions grew in 2012, according to a report in Britain’s Telegraph, when provincial authorities found ninety-nine identical copies of a single test.

  The next year, as students in Zhongxiang arrived to take the exam, they were dismayed to be funneled through metal detectors and forced to relinquish their mobile phones. Some surrendered tiny transmitters disguised as pencil erasers. Once inside, the students found themselves accompanied by fifty-four investigators from different school districts. A few of these investigators crossed the street to a hotel, where they found groups positioned to communicate with the students through their transmitters.

  The response to this crackdown on cheating was volcanic. Some two thousand stone-throwing protesters gathered in the street outside the school. They chanted, “We want fairness. There is no fairness if you don’t let us cheat.”

  It sounds like a joke, but they were absolutely serious. The stakes for the students were sky high. As they saw it, they faced a chance either to pursue an elite education and a prosperous career or to stay stuck in their provincial city, a relative backwater. And whether or not it was the case, they had the perception that others were cheating. So preventing the students in Zhongxiang from cheating was unfair. In a system in which cheating is the norm, following the rules amounts to a handicap. Just ask the Tour de France cyclists who were annihilated for seven years straight by Lance Armstrong and his doping teammates.

  The only way to win in such a scenario is to gain an advantage and to make sure that others aren’t getting a bigger one. This is the case not only in China but also in the United States, where high school admissions officers, parents, and students find themselves caught in a frantic effort to game the system spawned by the U.S. News model.

  An entire industry of coaches and tutors thrives on the model’s feedback loop and the anxiety it engenders. Many of them cost serious money. A four-day “application boot camp,” run by a company called Top Tier Admissions, costs $16,000 (plus room and board). During the sessions, the high school juniors develop their essays, learn how to “ace” their interviews, and create an “activity sheet” to sum up all the awards, sports, club activities, and community work that admissions officers are eager to see.

  Sixteen thousand dollars may sound like a lot of money. But much like the Chinese protesters in Zhongxiang, many American families fret that their children’s future success and fulfillment hinge upon acceptance to an elite university.

  The most effective coaches understand the admissions models at each college so that they can figure out how a potential student might fit into their portfolios. A California-based entrepreneur, Steven Ma, takes this market-based approach to an extreme. Ma, founder of ThinkTank Learning, places the prospective students into his own model and calculates the likelihood that they’ll get into their target colleges. He told Bloomberg BusinessWeek, for example, that an American-born senior with a 3.8 GPA, an SAT score of 2000, and eight hundred hours of extracurricular activities had a 20.4 percent shot of getting into New York University, and a 28.1 percent chance at the University of Southern California. ThinkTank then offers guaranteed consulting packages. If that hypothetical student follows the consultancy’s coaching and gets into NYU, it will cost $25,931, or $18,826 for USC. If he’s rejected, it costs nothing.

  Each college’s admissions model is derived, at least in part, from the U.S. News model, and each one is a mini-WMD. These models lead students and their parents to run in frantic circles and spend obscene amounts of money. And they’re opaque. This leaves most of the participants (or victims) in the dark. But it creates a big business for consultants, like Steven Ma, who manage to learn their secrets, either by cultivating sources at the universities or by reverse-engineering their algorithms.

  The victims, of course, are the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spent on courses and consultants. They miss out on precious insider knowledge. The result is an education system that favors the privileged. It tilts against needy students, locking out the great majority of them—and pushing them down a path toward poverty. It deepens the social divide.

  But even those who claw their way into a top college lose out. If you think about it, the college admissions game, while lucrative for some, has virtually no educational value. The complex and fraught production simply re-sorts and reranks the very same pool of eighteen-year-old kids in newfangled ways. They don’t master important skills by jumping through many more hoops or writing meticulously targeted college essays under the watchful eye of professional tutors. Others scrounge online for cut-rate versions of those tutors. All of them, from the rich to the working class, are simply being trained to fit into an enormous machine—to satisfy a WMD. And at the end of the ordeal, many of them will be saddled with debt that will take decades to pay off. They’re pawns in an arms race, and it’s a particularly nasty one.

  So is there a fix? During his second term, President Obama suggested coming up with a new college rankings model, one more in tune with national priorities and middle-class means than the U.S. News version. His secondary goal was to sap power from for-profit colleges (a money-sucking scourge that we’ll discuss in the next chapter). Obama’s idea would be to tie a college ranking system to a different set of metrics, including affordability, the percentage of poor and minority students, and postgraduation job placement. Like the U.S. News ranking, it would also consider graduation rate. If colleges dipped below the minimums in these categories, they’d get cut off from the $180 million-per-year federal student loan market (which the for-profit universities have been feasting on).

  All of those sound like worthy goals, to be sure, but every ranking system can be gamed. And when that happens, it creates new and different feedback loops and a host of unintended consequences.

  It’s easy to raise graduation rates, for example, by lowering standards. Many students struggle with math and science prerequisites and foreign languages. Water down those requirements, and more students will graduate. But if one goal of our educational system is to produce more scientists and technologists for a global economy, how smart is that? It would also be a cinch to pump up the income numbers for graduates. All colleges would have to do is shrink their liberal arts programs, and get rid of education departments and social work departments while they’re at it, since teachers and social workers make less money than engineers, chemists, and computer scientists. But they’re no less valuable to society.

  It also wouldn’t be too hard to lower costs. One approach already gaining popularity is to lower the percentage of tenured faculty, replacing these expensive professors, as they retire, with cheaper instructors, or adjuncts. For some departments at some universities, this might make sense. But there are costs. Tenured faculty, working with graduate students, power important research and set the standards for their departments, whereas harried adjuncts, who might teach five courses at three colleges just to pay rent, rarely have the time or energy to deliver more than commodity education. Another possible approach, that of removing unnecessary administrative positions, seems all too rare.

  The number of “graduates employed nine months after graduation” can be gamed too. A New York Times report in 2011 focused on law schools, which are already evaluated by their ability to position their students for careers. Say a newly minted lawyer with $150,000 in student loans is working as a barista. For some unscrupulous law schools investigated by the Times, he counted as employed. Some schools went further, hiring their own graduates for hourly temp jobs just as the crucial nine-month period approached. Others sent out surveys to recent alumni and counted all those that didn’t respond as “employed.”

  Perhaps it was just as well that the Obama administration failed to come up with a
rejiggered ranking system. The pushback by college presidents was fierce. After all, they had spent decades optimizing themselves to satisfy the U.S. News WMD. A new formula based on graduation rates, class size, alumni employment and income, and other metrics could wreak havoc with their ranking and reputation. No doubt they also made good points about the vulnerabilities of any new model and the new feedback loops it would generate.

  So the government capitulated. And the result might be better. Instead of a ranking, the Education Department released loads of data on a website. The result is that students can ask their own questions about the things that matter to them—including class size, graduation rates, and the average debt held by graduating students. They don’t need to know anything about statistics or the weighting of variables. The software itself, much like an online travel site, creates individual models for each person. Think of it: transparent, controlled by the user, and personal. You might call it the opposite of a WMD.

  One day during my stint as a data scientist for the advertising start-up Intent Media, a prominent venture capitalist visited the office. He seemed to be mulling an investment in the company, which was eager to put on its best face. So all of us were summoned to hear him speak.

  He outlined the brilliant future of targeted advertising. By contributing rivers of data, people would give advertisers the ability to learn about them in great detail. This would enable companies to target them with what they deemed valuable information, which would arrive at just the right time and place. A pizzeria, for example, might know that you’re not only in the neighborhood but also likely to be hungry for the same deep dish double cheese with pepperoni that you had last week at halftime of the Dallas Cowboys game. Their system might see that people whose data follows patterns similar to yours are more likely to click on a discount coupon during that twenty-minute window.

  The weakest part of his argument, it seemed to me, was its justification. He argued that the coming avalanche of personalized advertising would be so useful and timely that customers would welcome it. They would beg for more. As he saw it, most people objected to advertisements because they were irrelevant to them. In the future, they wouldn’t be. Presumably, folks in his exclusive demo would welcome pitches tailored to them, perhaps featuring cottages in the Bahamas, jars of hand-pressed virgin olive oil, or time-shares for private jets. And he joked that he would never have to see another ad for the University of Phoenix—a for-profit education factory that appeals largely to the striving (and more easily cheated) underclasses.

  It was strange, I thought, that he mentioned the University of Phoenix. Somehow he was seeing the ads, and I wasn’t. Or maybe I didn’t notice them. In any case, I knew quite a bit about for-profit universities, which had by that point become multimillion-dollar operations. These so-called diploma mills were often underwritten by government-financed loans, and the diplomas they awarded had scant value in the workplace. In many professions, they were no more valuable than a high school degree.

  While the WMD in the U.S. News Best Colleges ranking made life miserable for rich and middle-class students (and their families), the for-profit colleges focused on the other, more vulnerable, side of the population. And the Internet gave them the perfect tool to do so. It’s little surprise, therefore, that the industry’s dramatic growth coincided with the arrival of the Internet as an always-on communications platform for the masses. While spending more than $50 million on Google ads alone, the University of Phoenix targeted poor people with the bait of upward mobility. Its come-on carried the underlying criticism that the struggling classes weren’t doing enough to improve their lives. And it worked. Between 2004 and 2014, for-profit enrollment tripled, and the industry now accounts for 11 percent of the country’s college and university students.

  The marketing of these universities is a far cry from the early promise of the Internet as a great equalizing and democratizing force. If it was true during the early dot-com days that “nobody knows you’re a dog,” it’s the exact opposite today. We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns. This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises. They find inequality and feast on it. The result is that they perpetuate our existing social stratification, with all of its injustices. The greatest divide is between the winners in our system, like our venture capitalist, and the people his models prey upon.

  Anywhere you find the combination of great need and ignorance, you’ll likely see predatory ads. If people are anxious about their sex lives, predatory advertisers will promise them Viagra or Cialis, or even penis extensions. If they are short of money, offers will pour in for high-interest payday loans. If their computer is acting sludgy, it might be a virus inserted by a predatory advertiser, who will then offer to fix it. And as we’ll see, the boom in for-profit colleges is fueled by predatory ads.

  When it comes to WMDs, predatory ads practically define the genre. They zero in on the most desperate among us at enormous scale. In education, they promise what’s usually a false road to prosperity, while also calculating how to maximize the dollars they draw from each prospect. Their operations cause immense and nefarious feedback loops and leave their customers buried under mountains of debt. And the targets have little idea how they were scammed, because the campaigns are opaque. They just pop up on the computer, and later call on the phone. The victims rarely learn how they were chosen or how the recruiters came to know so much about them.

  Consider Corinthian College. Until recently, it was a giant in the industry. Its various divisions had more than eighty thousand students, the great majority of them receiving government-financed loans. In 2013, the for-profit college got busted by the attorney general of California for lying about job placement rates, overcharging students, and using unofficial military seals in predatory ads to reel in vulnerable people. The complaint pointed out that one of its divisions, Everest University Online’s Brandon Campus, charged $68,800 in tuition for an online bachelor’s degree in paralegal. (Such courses cost less than $10,000 at many traditional colleges around the country.)

  Moreover, according to the complaint, Corinthian College targeted “isolated,” “impatient” individuals with “low self esteem” who have “few people in their lives who care about them” and who are “stuck” and “unable to see and plan well for future.” The complaint called Corinthian College’s practices “unlawful, unfair, and fraudulent.” In 2014, amid more reports of abuses, the Obama administration put a hold on the company’s access to federal student loan funding. That was its lifeblood. In mid-2015, the company sold off most of its campuses and declared Chapter 11 bankruptcy.

  But the industry marches on. Vatterott College, a career-training institute, is a particularly nasty example. A 2012 Senate committee report on for-profit colleges described Vatterott’s recruiting manual, which sounds diabolical. It directs recruiters to target “Welfare Mom w/Kids. Pregnant Ladies. Recent Divorce. Low Self-Esteem. Low Income Jobs. Experienced a Recent Death. Physically/Mentally Abused. Recent Incarceration. Drug Rehabilitation. Dead-End Jobs—No Future.”

  Why, specifically, were they targeting these folks? Vulnerability is worth gold. It always has been. Picture an itinerant quack in an old western movie. He pulls into town with his wagon full of jangling jars and bottles. When he sits down with an elderly prospective customer, he seeks out her weaknesses. She covers her mouth when she smiles, indicating that she’s sensitive about her bad teeth. She anxiously twirls her old wedding ring, which from the looks of her swollen knuckle will be stuck there till the end of her days. Arthritis. So when he pitches his products to her, he focuses on the ugliness of her teeth and her aching hands. He can promise to restore the beauty of her smile and wash away the pain from her joints. With this knowledge, he knows he’s halfway to a sale before even clearing his throat to speak.

>   The playbook for predatory advertisers is similar, but they carry it out at massive scale, targeting millions of people every day. The customers’ ignorance, of course, is a crucial piece of the puzzle. Many of the targeted students are immigrants who come to this country believing that private universities are more prestigious than public ones. This argument is plausible if the private universities happen to be Harvard and Princeton. But the idea that DeVry or the University of Phoenix would be preferable to any state university (much less public gems such as Berkeley, Michigan, or Virginia) is something only newcomers to the system could ever believe.

  Once the ignorance is established, the key for the recruiter, just as for the snake-oil merchant, is to locate the most vulnerable people and then use their private information against them. This involves finding where they suffer the most, which is known as the “pain point.” It might be low self-esteem, the stress of raising kids in a neighborhood of warring gangs, or perhaps a drug addiction. Many people unwittingly disclose their pain points when they look for answers on Google or, later, when they fill out college questionnaires. With that valuable nugget in hand, recruiters simply promise that an expensive education at their university will provide the solution and eliminate the pain. “We deal with people that live in the moment and for the moment,” Vatterott’s training materials explain. “Their decision to start, stay in school or quit school is based more on emotion than logic. Pain is the greater motivator in the short term.” A recruiting team at ITT Technical Institute went so far as to draw up an image of a dentist bearing down on a patient in agony, with the words “Find Out Where Their Pain Is.”