Homo Deus

by Yuval Noah Harari

  From Oracle to Sovereign

  Once Google, Facebook and other algorithms become all-knowing oracles, they may well evolve into agents and finally into sovereigns.33 To understand this trajectory, consider the case of Waze – a GPS-based navigational application which many drivers use nowadays. Waze isn’t just a map. Its millions of users constantly update it about traffic jams, car accidents and police cars. Hence Waze knows to divert you away from heavy traffic, and bring you to your destination through the quickest possible route. When you reach a junction and your gut instinct tells you to turn right, but Waze instructs you to turn left, users sooner or later learn that they had better listen to Waze rather than to their feelings.34

  At first sight it seems that the Waze algorithm serves us only as an oracle. We ask a question, the oracle replies, but it is up to us to make a decision. If the oracle wins our trust, however, the next logical step is to turn it into an agent. We give the algorithm only a final aim, and it acts to realise that aim without our supervision. In the case of Waze, this may happen when we connect Waze to a self-driving car, and tell Waze ‘take the fastest route home’ or ‘take the most scenic route’ or ‘take the route which will result in the minimum amount of pollution’. We call the shots, but leave it to Waze to execute our commands.

  Finally, Waze might become sovereign. Having so much power in its hands, and knowing far more than we know, it may start manipulating us, shaping our desires and making our decisions for us. For example, suppose because Waze is so good, everybody starts using it. And suppose there is a traffic jam on route no. 1, while the alternative route no. 2 is relatively open. If Waze simply lets everybody know that, then all drivers will rush to route no. 2, and it too will be clogged. When everybody uses the same oracle, and everybody believes the oracle, the oracle turns into a sovereign. So Waze must think for us. Maybe it will inform only half the drivers that route no. 2 is open, while keeping this information secret from the other half. Thereby pressure will ease on route no. 1 without blocking route no. 2.

  Microsoft is developing a far more sophisticated system called Cortana, named after an AI character in their popular Halo videogame series. Cortana is an AI personal assistant which Microsoft hopes to include as an integral feature of future versions of Windows. Users will be encouraged to allow Cortana access to all their files, emails and applications, so that it will get to know them, and can offer its advice on myriad matters, as well as becoming a virtual agent representing the user’s interests. Cortana could remind you to buy something for your wife’s birthday, select the present, reserve a table at the restaurant and prompt you to take your medicine an hour before dinner. It could alert you that if you don’t stop reading now, you will be late for an important business meeting. As you are about to enter the meeting, Cortana will warn that your blood pressure is too high and your dopamine level too low, and based on past statistics, you tend to make serious business mistakes in such circumstances. So you had better keep things tentative and avoid committing yourself or signing any deals.

  Once Cortanas evolve from oracles to agents, they might start speaking directly with one another, on their masters’ behalf. It can begin innocently enough, with my Cortana contacting your Cortana to agree on a place and time for a meeting. Next thing I know, a potential employer tells me not to bother sending a CV, but simply allow his Cortana to grill my Cortana. Or my Cortana may be approached by the Cortana of a potential lover, and the two will compare notes to decide whether it’s a good match – completely unbeknown to their human owners.

  As Cortanas gain authority, they may begin manipulating each other to further the interests of their masters, so that success in the job market or the marriage market may increasingly depend on the quality of your Cortana. Rich people owning the most up-to-date Cortana will have a decisive advantage over poor people with their older versions.

  But the murkiest issue of all concerns the identity of Cortana’s master. As we have seen, humans are not individuals, and they don’t have a single unified self. Whose interests, then, should Cortana serve? Suppose my narrating self makes a New Year resolution to start a diet and go to the gym every day. A week later, when it is time to go to the gym, the experiencing self asks Cortana to turn on the TV and order pizza. What should Cortana do? Should it obey the experiencing self, or the resolution taken a week ago by the narrating self?

  You may well ask whether Cortana is really different from an alarm clock, which the narrating self sets in the evening, in order to wake the experiencing self in time for work. But Cortana will have far more power over me than an alarm clock. The experiencing self can silence the alarm clock by pressing a button. In contrast, Cortana will know me so well that it will know exactly what inner buttons to push in order to make me follow its ‘advice’.

  Microsoft’s Cortana is not alone in this game. Google Now and Apple’s Siri are headed in the same direction. Amazon too has algorithms that constantly study you and use their knowledge to recommend products. When I go to Amazon to buy a book, an ad pops up and tells me: ‘I know which books you liked in the past. People with similar tastes also tend to love this or that new book.’ Wonderful! There are millions of books in the world, and I can never go over all of them, not to mention predicting accurately which ones I would like. How good that an algorithm knows me, and can give me recommendations based on my unique taste.

  And this is just the beginning. Today in the US more people read digital books than printed volumes. Devices such as Amazon’s Kindle are able to collect data on their users while they are reading the book. For example, your Kindle can monitor which parts of the book you read fast, and which slow; on which page you took a break, and on which sentence you abandoned the book, never to pick it up again. (Better tell the author to rewrite that bit.) If Kindle is upgraded with face recognition and biometric sensors, it can know what made you laugh, what made you sad and what made you angry. Soon, books will read you while you are reading them. And whereas you quickly forget most of what you read, Amazon will never forget a thing. Such data will enable Amazon to evaluate the suitability of a book much better than ever before. It will also enable Amazon to know exactly who you are, and how to turn you on and off.35

  Eventually, we may reach a point when it will be impossible to disconnect from this all-knowing network even for a moment. Disconnection will mean death. If medical hopes are realised, future people will incorporate into their bodies a host of biometric devices, bionic organs and nano-robots, which will monitor our health and defend us from infections, illnesses and damage. Yet these devices will have to be online 24/7, both in order to be updated with the latest medical news, and in order to protect them from the new plagues of cyberspace. Just as my home computer is constantly attacked by viruses, worms and Trojan horses, so will be my pacemaker, my hearing aid and my nanotech immune system. If I don’t update my body’s anti-virus program regularly, I will wake up one day to discover that the millions of nano-robots coursing through my veins are now controlled by a North Korean hacker.

  The new technologies of the twenty-first century may thus reverse the humanist revolution, stripping humans of their authority, and empowering non-human algorithms instead. If you are horrified by this direction, don’t blame the computer geeks. The responsibility actually lies with the biologists. It is crucial to realise that this entire trend is fuelled by biological insights more than by computer science. It is the life sciences that have concluded that organisms are algorithms. If this is not the case – if organisms function in an inherently different way to algorithms – then computers may work wonders in other fields, but they will not be able to understand us and direct our life, and they will certainly be incapable of merging with us. Yet once biologists concluded that organisms are algorithms, they dismantled the wall between the organic and inorganic, turned the computer revolution from a purely mechanical affair into a biological cataclysm, and shifted authority from individual humans to networked algorithms.

 
Some people are indeed horrified by this development, but the fact is that millions willingly embrace it. Already today many of us give up our privacy and our individuality, record our every action, conduct our lives online and become hysterical if connection to the net is interrupted even for a few minutes. The shifting of authority from humans to algorithms is happening all around us, not as a result of some momentous governmental decision, but due to a flood of mundane choices.

  The result will not be an Orwellian police state. We always prepare ourselves for the previous enemy, even when we face an altogether new menace. Defenders of human individuality stand guard against the tyranny of the collective, without realising that human individuality is now threatened from the opposite direction. The individual will not be crushed by Big Brother; it will disintegrate from within. Today corporations and governments pay homage to my individuality, and promise to provide medicine, education and entertainment customised to my unique needs and wishes. But in order to so, corporations and governments first need to break me up into biochemical subsystems, monitor these subsystems with ubiquitous sensors and decipher their working with powerful algorithms. In the process, the individual will transpire to be nothing but a religious fantasy. Reality will be a mesh of biochemical and electronic algorithms, without clear borders, and without individual hubs.

  Upgrading Inequality

  So far we have looked at two of the three practical threats to liberalism: firstly, that humans will lose their value completely; secondly, that humans will still be valuable collectively, but they will lose their individual authority, and will instead be managed by external algorithms. The system will still need you to compose symphonies, teach history or write computer code, but the system will know you better than you know yourself, and will therefore make most of the important decisions for you – and you will be perfectly happy with that. It won’t necessarily be a bad world; it will, however, be a post-liberal world.

  The third threat to liberalism is that some people will remain both indispensable and undecipherable, but they will constitute a small and privileged elite of upgraded humans. These superhumans will enjoy unheard-of abilities and unprecedented creativity, which will allow them to go on making many of the most important decisions in the world. They will perform crucial services for the system, while the system could not understand and manage them. However, most humans will not be upgraded, and they will consequently become an inferior caste, dominated by both computer algorithms and the new superhumans.

  Splitting humankind into biological castes will destroy the foundations of liberal ideology. Liberalism can coexist with socio-economic gaps. Indeed, since it favours liberty over equality, it takes such gaps for granted. However, liberalism still presupposes that all human beings have equal value and authority. From a liberal perspective, it is perfectly all right that one person is a billionaire living in a sumptuous chateau, whereas another is a poor peasant living in a straw hut. For according to liberalism, the peasant’s unique experiences are still just as valuable as the billionaire’s. That’s why liberal authors write long novels about the experiences of poor peasants – and why even billionaires read such books avidly. If you go to see Les Misérables on Broadway or in Covent Garden, you will find that good seats can cost hundreds of dollars, and the audience’s combined wealth probably runs into the billions, yet they still sympathise with Jean Valjean who served nineteen years in jail for stealing a loaf of bread to feed his starving nephews.

  The same logic operates on election day, when the vote of the poor peasant counts for exactly the same as the billionaire’s. The liberal solution for social inequality is to give equal value to different human experiences, instead of trying to create the same experiences for everyone. However, what will be the fate of this solution once rich and poor are separated not merely by wealth, but also by real biological gaps?

  In her New York Times article, Angelina Jolie referred to the high costs of genetic testing. At present, the test Jolie had taken costs $3,000 (which does not include the price of the actual mastectomy, the reconstruction surgery and related treatments). This in a world where 1 billion people earn less than $1 per day, and another 1.5 billion earn between $1 and $2 a day.36 Even if they work hard their entire life, they will never be able to finance a $3,000 genetic test. And the economic gaps are at present only increasing. As of early 2016, the sixty-two richest people in the world were worth as much as the poorest 3.6 billion people! Since the world’s population is about 7.2 billion, it means that these sixty-two billionaires together hold as much wealth as the entire bottom half of humankind.37

  The cost of DNA testing is likely to go down with time, but expensive new procedures are constantly being pioneered. So while old treatments will gradually come within reach of the masses, the elites will always remain a couple of steps ahead. Throughout history the rich enjoyed many social and political advantages, but there was never a huge biological gap separating them from the poor. Medieval aristocrats claimed that superior blue blood was flowing through their veins, and Hindu Brahmins insisted that they were naturally smarter than everyone else, but this was pure fiction. In the future, however, we may see real gaps in physical and cognitive abilities opening between an upgraded upper class and the rest of society.

  When scientists are confronted with this scenario, their standard reply is that in the twentieth century too many medical breakthroughs began with the rich, but eventually benefited the whole population and helped to narrow rather than widen the social gaps. For example, vaccines and antibiotics at first profited mainly the upper classes in Western countries, but today they improve the lives of all humans everywhere.

  However, the expectation that this process will be repeated in the twenty-first century may be just wishful thinking, for two important reasons. First, medicine is undergoing a tremendous conceptual revolution. Twentieth-century medicine aimed to heal the sick. Twenty-first-century medicine is increasingly aiming to upgrade the healthy. Healing the sick was an egalitarian project, because it assumed that there is a normative standard of physical and mental health that everyone can and should enjoy. If someone fell below the norm, it was the job of doctors to fix the problem and help him or her ‘be like everyone’. In contrast, upgrading the healthy is an elitist project, because it rejects the idea of a universal standard applicable to all, and seeks to give some individuals an edge over others. People want superior memories, above-average intelligence and first-class sexual abilities. If some form of upgrade becomes so cheap and common that everyone enjoys it, it will simply be considered the new baseline, which the next generation of treatments will strive to surpass.

  Second, twentieth-century medicine benefited the masses because the twentieth century was the age of the masses. Twentieth-century armies needed millions of healthy soldiers, and the economy needed millions of healthy workers. Consequently, states established public health services to ensure the health and vigour of everyone. Our greatest medical achievements were the provision of mass-hygiene facilities, the campaigns of mass vaccinations and the overcoming of mass epidemics. The Japanese elite in 1914 had a vested interest in vaccinating the poor and building hospitals and sewage systems in the slums, because if they wanted Japan to be a strong nation with a strong army and a strong economy, they needed many millions of healthy soldiers and workers.

  But the age of the masses may be over, and with it the age of mass medicine. As human soldiers and workers give way to algorithms, at least some elites may conclude that there is no point in providing improved or even standard conditions of health for masses of useless poor people, and it is far more sensible to focus on upgrading a handful of superhumans beyond the norm.

  Already today, the birth rate is falling in technologically advanced countries such as Japan and South Korea, where prodigious efforts are invested in the upbringing and education of fewer and fewer children – from whom more and more is expected. How could huge developing countries like India, Brazil or Nigeria hope to com
pete with Japan? These countries resemble a long train. The elites in the first-class carriages enjoy health care, education and income levels on a par with the most developed nations in the world. However, the hundreds of millions of ordinary citizens who crowd the third-class carriages still suffer from widespread diseases, ignorance and poverty. What would the Indian, Brazilian or Nigerian elites prefer to do in the coming century? Invest in fixing the problems of hundreds of millions of poor, or in upgrading a few million rich? Unlike in the twentieth century, when the elite had a stake in fixing the problems of the poor because they were militarily and economically vital, in the twenty-first century the most efficient (albeit ruthless) strategy may be to let go of the useless third-class carriages, and dash forward with the first class only. In order to compete with Japan, Brazil might need a handful of upgraded superhumans far more than millions of healthy ordinary workers.

  How can liberal beliefs survive the appearance of superhumans with exceptional physical, emotional and intellectual abilities? What will happen if it turns out that such superhumans have fundamentally different experiences to normal Sapiens? What if superhumans are bored by novels about the experiences of lowly Sapiens thieves, whereas run-of-the-mill humans find soap operas about superhuman love affairs unintelligible?

  The great human projects of the twentieth century – overcoming famine, plague and war – aimed to safeguard a universal norm of abundance, health and peace for all people without exception. The new projects of the twenty-first century – gaining immortality, bliss and divinity – also hope to serve the whole of humankind. However, because these projects aim at surpassing rather than safeguarding the norm, they may well result in the creation of a new superhuman caste that will abandon its liberal roots and treat normal humans no better than nineteenth-century Europeans treated Africans.