Through machine-learning techniques like decision trees and neural networks, which dynamically model complex statistical relationships among phenomena, computers are also able to refine the way they make predictions as they process more data and receive feedback about the accuracy of earlier guesses.38 The weightings they give different variables get more precise, and their calculations of probability better reflect what happens in the real world. Today’s computers get smarter as they gain experience, just as people do. New “neuromorphic” microchips, which have machine-learning protocols hardwired into their circuitry, will boost computers’ learning ability in coming years, some computer scientists believe. Machines will become more discerning. We may bristle at the idea that computers are “smart” or “intelligent,” but the fact is that while they may lack the understanding, empathy, and insight of doctors, computers are able to replicate many of the judgments of doctors through the statistical analysis of large amounts of digital information—what’s come to be known as “big data.” Many of the old debates about the meaning of intelligence are being rendered moot by the brute number-crunching force of today’s data-processing machines.
The diagnostic skills of computers will only get better. As more data about individual patients are collected and stored, in the form of electronic records, digitized images and test results, pharmacy transactions, and, in the not-too-distant future, readings from personal biological sensors and health-monitoring apps, computers will become more proficient at finding correlations and calculating probabilities at ever finer levels of detail. Templates and guidelines will become more comprehensive and elaborate. Given the current stress on achieving greater efficiency in health care, we’re likely to see the Taylorist ethos of optimization and standardization take hold throughout the medical field. The already strong trend toward replacing personal clinical judgment with the statistical outputs of so-called evidence-based medicine will gain momentum. Doctors will face increasing pressure, if not outright managerial fiat, to cede more control over diagnoses and treatment decisions to software.
To put it into uncharitable but not inaccurate terms, many doctors may soon find themselves taking on the role of human sensors who collect information for a decision-making computer. The doctors will examine the patient and enter data into electronic forms, but the computer will take the lead in suggesting diagnoses and recommending therapies. Thanks to the steady escalation of computer automation through Bright’s hierarchy, physicians seem destined to experience, at least in some areas of their practice, the same deskilling effect that was once restricted to factory hands.
They will not be alone. The incursion of computers into elite professional work is happening everywhere. We’ve already seen how the thinking of corporate auditors is being shaped by expert systems that make predictions about risks and other variables. Other financial professionals, from loan officers to investment managers, also depend on computer models to guide their decisions, and Wall Street is now largely under the control of correlation-sniffing computers and the quants who program them. The number of people employed as securities dealers and traders in New York City plummeted by a third, from 150,000 to 100,000, between 2000 and 2013, despite the fact that Wall Street firms were often posting record profits. The overriding goal of brokerage and investment banking firms is “automating the system and getting rid of the traders,” one financial industry analyst explained to a Bloomberg reporter. As for the traders who remain, “all they do today is hit buttons on computer screens.”39
That’s true not only in the trading of simple stocks and bonds but also in the packaging and dealing of complex financial instruments. Ashwin Parameswaran, a technology analyst and former investment banker, notes that “banks have made a significant effort to reduce the amount of skill and know-how required to price and trade financial derivatives. Trading systems have been progressively modified so that as much knowledge as possible is embedded within the software.” 40 Predictive algorithms are even moving into the lofty realm of venture capitalism, where top investors have long prided themselves on having a good nose for business and innovation. Prominent venture-capital firms like the Ironstone Group and Google Ventures now use computers to sniff out patterns in records of entrepreneurial success, and they place their bets accordingly.
A similar trend is under way in the law. For years, attorneys have depended on computers to search legal databases and prepare documents. Recently, software has taken a more central role in law offices. The critical process of document discovery, in which, traditionally, junior lawyers and paralegals read through reams of correspondence, email messages, and notes in search of evidence, has been largely automated. Computers can parse thousands of pages of digitized documents in seconds. Using e-discovery software with language-analysis algorithms, the machines not only spot relevant words and phrases but also discern chains of events, relationships among people, and even personal emotions and motivations. A single computer can take over the work of dozens of well-paid professionals. Document-preparation software has also advanced. By filling out a simple checklist, a lawyer can assemble a complex contract in an hour or two—a job that once took days.
On the horizon are bigger changes. Legal software firms are beginning to develop statistical prediction algorithms that, by analyzing many thousands of past cases, can recommend trial strategies, such as the choice of a venue or the terms of a settlement offer, that carry high probabilities of success. Software will soon be able to make the kinds of judgments that up to now required the experience and insight of a senior litigator.41 Lex Machina, a company started in 2010 by a group of Stanford law professors and computer scientists, offers a preview of what’s coming. With a database covering some 150,000 intellectual property cases, it runs computer analyses that predict the outcomes of patent lawsuits under various scenarios, taking into account the court, the presiding judge and participating attorneys, the litigants, the outcomes of related cases, and other factors.
Predictive algorithms are also assuming more control over the decisions made by business executives. Companies are spending billions of dollars a year on “people analytics” software that automates decisions about hiring, pay, and promotion. Xerox now relies exclusively on computers to choose among applicants for its fifty thousand call-center jobs. Candidates sit at a computer for a half-hour personality test, and the hiring software immediately gives them a score reflecting the likelihood that they’ll perform well, show up for work reliably, and stick with the job. The company extends offers to those with high scores and sends low scorers on their way.42 UPS uses predictive algorithms to chart daily routes for its drivers. Retailers use them to determine the optimal arrangement of merchandise on store shelves. Marketers and ad agencies use them in deciding where and when to run advertisements and in generating promotional messages on social networks. Managers increasingly find themselves playing a subservient role to software. They review and rubber-stamp plans and decisions produced by computers.
There’s an irony here. In shifting the center of the economy from physical goods to data flows, computers brought new status and wealth to information workers during the last decades of the twentieth century. People who made their living by manipulating signs and symbols on screens became the stars of the new economy, even as the factory jobs that had long buttressed the middle class were being transferred overseas or handed off to robots. The dot-com bubble of the late 1990s, when for a few euphoric years riches flooded out of computer networks and into personal brokerage accounts, seemed to herald the start of a golden age of unlimited economic opportunity—what technology boosters dubbed a “long boom.” But the good times proved fleeting. Now we’re seeing that, as Norbert Wiener predicted, automation doesn’t play favorites. Computers are as good at analyzing symbols and otherwise parsing and managing information as they are at directing the moves of industrial robots. Even the people who operate complex computer systems are losing their jobs to software, as data centers, like factories, bec
ome increasingly automated. The vast server farms operated by companies like Google, Amazon, and Apple essentially run themselves. Thanks to virtualization, an engineering technique that uses software to replicate the functions of hardware components like servers, the facilities’ operations can be monitored and controlled by algorithms. Network problems and application glitches can be detected and fixed automatically, often in a matter of seconds. It may turn out that the late twentieth century’s “intellectualization of labor,” as the Italian media scholar Franco Berardi has termed it,43 was just a precursor to the early twenty-first century’s automation of intellect.
It’s always risky to speculate how far computers will go in mimicking the insights and judgments of people. Extrapolations based on recent computing trends have a way of turning into fantasies. But even if we assume, contrary to the extravagant promises of big-data evangelists, that there are limits to the applicability and usefulness of correlation-based predictions and other forms of statistical analysis, it seems clear that computers are a long way from bumping up against those limits. When, in early 2011, the IBM supercomputer Watson took the crown as the reigning champion of Jeopardy!, thrashing two of the quiz show’s top players, we got a preview of where computers’ analytical talents are heading. Watson’s ability to decipher clues was astonishing, but by the standards of contemporary artificial-intelligence programming, the computer was not performing an exceptional feat. It was, essentially, searching a vast database of documents for potential answers and then, by working simultaneously through a variety of prediction routines, determining which answer had the highest probability of being the correct one. But it was performing that feat so quickly that it was able to outthink exceptionally smart people in a tricky test involving trivia, wordplay, and recall.
Watson represents the flowering of a new, pragmatic form of artificial intelligence. Back in the 1950s and 1960s, when digital computers were still new, many mathematicians and engineers, and quite a few psychologists and philosophers, came to believe that the human brain had to operate like some sort of digital calculating machine. They saw in the computer a metaphor and a model for the mind. Creating artificial intelligence, it followed, would be fairly straightforward: you’d figure out the algorithms that run inside our skulls and then you’d translate those programs into software code. It didn’t work. The original artificial-intelligence strategy failed miserably. Whatever it is that goes on inside our brains, it turned out, can’t be reduced to the computations that go on inside computers.* Today’s computer scientists are taking a very different approach to artificial intelligence that’s at once less ambitious and more effective. The goal is no longer to replicate the process of human thought—that’s still beyond our ken—but rather to replicate its results. These scientists look at a particular product of the mind—a hiring decision, say, or an answer to a trivia question—and then program a computer to accomplish the same result in its own mindless way. The workings of Watson’s circuits bear little resemblance to the workings of the mind of a person playing Jeopardy!, but Watson can still post a higher score.
In the 1930s, while working on his doctoral thesis, the British mathematician and computing pioneer Alan Turing came up with the idea of an “oracle machine.” It was a kind of computer that, applying a set of explicit rules to a store of data through “some unspecified means,” could answer questions that normally would require tacit human knowledge. Turing was curious to figure out “how far it is possible to eliminate intuition, and leave only ingenuity.” For the purposes of his thought experiment, he posited that there would be no limit to the machine’s number-crunching acumen, no upper bound to the speed of its calculations or the amount of data it could take into account. “We do not mind how much ingenuity is required,” he wrote, “and therefore assume it to be available in unlimited supply.” 44 Turing was, as usual, prescient. He understood, as few others did at the time, the latent intelligence of algorithms, and he foresaw how that intelligence would be released by speedy calculations. Computers and databases will always have limits, but in systems like Watson we see the arrival of operational oracle machines. What Turing could only imagine, engineers are now building. Ingenuity is replacing intuition.
Watson’s data-analysis acumen is being put to practical use as a diagnostic aid for oncologists and other doctors, and IBM foresees further applications in such fields as law, finance, and education. Spy agencies like the CIA and the NSA are also reported to be testing the system. If Google’s driverless car reveals the newfound power of computers to replicate our psychomotor skills, to match or exceed our ability to navigate the physical world, Watson demonstrates computers’ newfound power to replicate our cognitive skills, to match or exceed our ability to navigate the world of symbols and ideas.
BUT THE replication of the outputs of thinking is not thinking. As Turing himself stressed, algorithms will never replace intuition entirely. There will always be a place for “spontaneous judgments which are not the result of conscious trains of reasoning.” 45 What really makes us smart is not our ability to pull facts from documents or decipher statistical patterns in arrays of data. It’s our ability to make sense of things, to weave the knowledge we draw from observation and experience, from living, into a rich and fluid understanding of the world that we can then apply to any task or challenge. It’s this supple quality of mind, spanning conscious and unconscious cognition, reason and inspiration, that allows human beings to think conceptually, critically, metaphorically, speculatively, wittily—to take leaps of logic and imagination.
Hector Levesque, a computer scientist and roboticist at the University of Toronto, provides an example of a simple question that people can answer in a snap but that baffles computers:
The large ball crashed right through the table because it was made of Styrofoam.
What was made of Styrofoam, the large ball or the table?
We come up with the answer effortlessly because we understand what Styrofoam is and what happens when you drop something on a table and what tables tend to be like and what the adjective large implies. We grasp the context, both of the situation and of the words used to describe it. A computer, lacking any true understanding of the world, finds the language of the question hopelessly ambiguous. It remains locked in its algorithms. Reducing intelligence to the statistical analysis of large data sets “can lead us,” says Levesque, “to systems with very impressive performance that are nonetheless idiot-savants.” They might be great at chess or Jeopardy! or facial recognition or other tightly circumscribed mental exercises, but they “are completely hopeless outside their area of expertise.” 46 Their precision is remarkable, but it’s often a symptom of the narrowness of their perception.
Even when aimed at questions amenable to probabilistic answers, computer analysis is not flawless. The speed and apparent exactitude of computer calculations can mask limitations and distortions in the underlying data, not to mention imperfections in the data-mining algorithms themselves. Any large data set holds an abundance of spurious correlations along with the reliable ones. It’s not hard to be misled by mere coincidence or to conjure a phantom association.47 Once a particular data set becomes the basis for important decisions, moreover, the data and its analysis become vulnerable to corruption. Seeking financial, political, or social advantage, people will try to game the system. As the social scientist Donald T. Campbell explained in a renowned 1976 paper, “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” 48
Flaws in data and algorithms can leave professionals, and the rest of us, susceptible to an especially pernicious form of automation bias. “The threat is that we will let ourselves be mindlessly bound by the output of our analyses even when we have reasonable grounds for suspecting something is amiss,” warn Viktor Mayer-Schönberger and Kenneth Cukier in their 2013 book Big Data. �
��Or that we will attribute a degree of truth to the data which it does not deserve.” 49 A particular risk with correlation-calculating algorithms stems from their reliance on data about the past to anticipate the future. In most cases, the future behaves as expected; it follows precedent. But on those peculiar occasions when conditions veer from established patterns, the algorithms can make wildly inaccurate predictions—a fact that has already spelled disaster for some highly computerized hedge funds and brokerage firms. For all their gifts, computers still display a frightening lack of common sense.
The more we embrace what Microsoft researcher Kate Crawford terms “data fundamentalism,”50 the more tempted we’ll be to devalue the many talents computers can’t mimic—to grant so much control to software that we restrict people’s ability to exercise the know-how that comes from real experience and that can often lead to creative, counterintuitive insights. As some of the unforeseen consequences of electronic medical records show, templates and formulas are necessarily reductive and can all too easily become straightjackets of the mind. The Vermont doctor and medical professor Lawrence Weed has, since the 1960s, been a forceful and eloquent advocate for using computers to help doctors make smart, informed decisions.51 He’s been called the father of electronic medical records. But even he warns that the current “misguided use of statistical knowledge” in medicine “systematically excludes the individualized knowledge and data essential to patient care.”52
The Glass Cage: Automation and Us Page 12