by Ted Chiang
I don’t know if humans will ever have that kind of relationship with AIs, but I feel like this is an area that’s been largely overlooked in science fiction. I’ve read a lot of stories in which people argue that AIs deserve legal rights, but in focusing on the big philosophical question, there’s a mundane reality that these stories gloss over. It’s a bit like how movies show separated lovers overcoming tremendous obstacles to be reunited: that’s wonderfully romantic, but it’s not the whole story when it comes to love; over the long term, love also means working through money problems and picking dirty laundry off the floor. So while achieving legal rights for AIs would clearly be a major milestone, another stage that I think is just as important—and indeed, probably a prerequisite for initiating a legal battle—is for people to put real effort into their individual relationships with AIs.
And even if we don’t care about them having legal rights, there’s still good reason to treat AIs with respect. Think about the pets of neglectful owners, or the lovers who have never stayed with someone longer than a month; are they the pets or lovers you would choose? Think about the kind of people that bad parenting produces; are those the people you want to have as your friends or your employees? No matter what roles we assign AIs, I suspect they will do a better job if, at some point during their development, there were people who cared about them.
What's Expected of Us
It's a tough choice...
This is a warning. Please read carefully.
By now you've probably seen a Predictor; millions of them have been sold by the time you're reading this. For those who haven't seen one, it's a small device, like a remote for opening your car door. Its only features are a button and a big green LED. The light flashes if you press the button. Specifically, the light flashes one second before you press the button.
Most people say that when they first try it, it feels like they're playing a strange game, one where the goal is to press the button after seeing the flash, and it's easy to play. But when you try to break the rules, you find that you can't. If you try to press the button without having seen a flash, the flash immediately appears, and no matter how fast you move, you never push the button until a second has elapsed. If you wait for the flash, intending to keep from pressing the button afterwards, the flash never appears. No matter what you do, the light always precedes the button press. There's no way to fool a Predictor.
The heart of each Predictor is a circuit with a negative time delay — it sends a signal back in time. The full implications of the technology will become apparent later, when negative delays of greater than a second are achieved, but that's not what this warning is about. The immediate problem is that Predictors demonstrate that there's no such thing as free will.
There have always been arguments showing that free will is an illusion, some based on hard physics, others based on pure logic. Most people agree these arguments are irrefutable, but no one ever really accepts the conclusion. The experience of having free will is too powerful for an argument to overrule. What it takes is a demonstration, and that's what a Predictor provides.
Typically, a person plays with a Predictor compulsively for several days, showing it to friends, trying various schemes to outwit the device. The person may appear to lose interest in it, but no one can forget what it means — over the following weeks, the implications of an immutable future sink in. Some people, realizing that their choices don't matter, refuse to make any choices at all. Like a legion of Bartleby the Scriveners, they no longer engage in spontaneous action. Eventually, a third of those who play with a Predictor must be hospitalized because they won't feed themselves. The end state is akinetic mutism, a kind of waking coma. They'll track motion with their eyes, and change position occasionally, but nothing more. The ability to move remains, but the motivation is gone.
Before people started playing with Predictors, akinetic mutism was very rare, a result of damage to the anterior cingulate region of the brain. Now it spreads like a cognitive plague. People used to speculate about a thought that destroys the thinker, some unspeakable lovecraftian horror, or a Gödel sentence that crashes the human logical system. It turns out that the disabling thought is one that we've all encountered: the idea that free will doesn't exist. It just wasn't harmful until you believed it.
Doctors try arguing with the patients while they still respond to conversation. We had all been living happy, active lives before, they reason, and we hadn't had free will then either. Why should anything change? "No action you took last month was any more freely chosen than one you take today," a doctor might say. "You can still behave that way now." The patients invariably respond, "But now I know." And some of them never say anything again.
Some will argue that the fact the Predictor causes this change in behaviour means that we do have free will. An automaton cannot become discouraged, only a free-thinking entity can. The fact that some individuals descend into akinetic mutism whereas others do not just highlights the importance of making a choice.
Unfortunately, such reasoning is faulty: every form of behaviour is compatible with determinism. One dynamic system might fall into a basin of attraction and wind up at a fixed point, whereas another exhibits chaotic behaviour indefinitely, but both are completely deterministic.
I'm transmitting this warning to you from just over a year in your future: it's the first lengthy message received when circuits with negative delays in the megasecond range are used to build communication devices. Other messages will follow, addressing other issues. My message to you is this: pretend that you have free will. It's essential that you behave as if your decisions matter, even though you know that they don't. The reality isn't important: what's important is your belief, and believing the lie is the only way to avoid a waking coma. Civilization now depends on self-deception. Perhaps it always has.
And yet I know that, because free will is an illusion, it's all predetermined who will descend into akinetic mutism and who won't. There's nothing anyone can do about it — you can't choose the effect the Predictor has on you. Some of you will succumb and some of you won't, and my sending this warning won't alter those proportions. So why did I do it?
Because I had no choice.
The Merchant and the Alchemist’s Gate
O mighty Caliph and Commander of the Faithful, I am humbled to be in the splendor of your presence; a man can hope for no greater blessing as long as he lives. The story I have to tell is truly a strange one, and were the entirety to be tattooed at the corner of one’s eye, the marvel of its presentation would not exceed that of the events recounted, for it is a warning to those who would be warned and a lesson to those who would learn.
My name is Fuwaad ibn Abbas, and I was born here in Baghdad, City of Peace. My father was a grain merchant, but for much of my life I have worked as a purveyor of fine fabrics, trading in silk from Damascus and linen from Egypt and scarves from Morocco that are embroidered with gold. I was prosperous, but my heart was troubled, and neither the purchase of luxuries nor the giving of alms were able to soothe it. Now I stand before you without a single dirham in my purse, but I am at peace.
Allah is the beginning of all things, but with Your Majesty’s permission, I begin my story with the day I took a walk through the district of metalsmiths. I needed to purchase a gift for a man I had to do business with, and had been told he might appreciate a tray made of silver. After browsing for half an hour, I noticed that one of the largest shops in the market had been taken over by a new merchant. It was a prized location that must have been expensive to acquire, so I entered to peruse its wares.
Never before had I seen such a marvelous assortment of goods. Near the entrance there was an astrolabe equipped with seven plates inlaid with silver, a water-clock that chimed on the hour, and a nightingale made of brass that sang when the wind blew. Further inside there were even more ingenious mechanisms, and I stared at them the way a child watches a juggler, when an old man stepped out from a doorway in the ba
ck.
“Welcome to my humble shop, my lord,” he said. “My name is Bashaarat. How may I assist you?”
“These are remarkable items that you have for sale. I deal with traders from every corner of the world, and yet I have never seen their like. From where, may I ask, did you acquire your merchandise?”
“I am grateful to you for your kind words,” he said. “Everything you see here was made in my workshop, by myself or by my assistants under my direction.”
I was impressed that this man could be so well versed in so many arts. I asked him about the various instruments in his shop, and listened to him discourse learnedly about astrology, mathematics, geomancy, and medicine. We spoke for over an hour, and my fascination and respect bloomed like a flower warmed by the dawn, until he mentioned his experiments in alchemy.
“Alchemy?” I said. This surprised me, for he did not seem the type to make such a sharper’s claim. “You mean you can turn base metal into gold?”
“I can, my lord, but that is not in fact what most seek from alchemy.”
“What do most seek, then?”
“They seek a source of gold that is cheaper than mining ore from the ground. Alchemy does describe a means to make gold, but the procedure is so arduous that, by comparison, digging beneath a mountain is as easy as plucking peaches from a tree.”
I smiled. “A clever reply. No one could dispute that you are a learned man, but I know better than to credit alchemy.”
Bashaarat looked at me and considered. “I have recently built something that may change your opinion. You would be the first person I have shown it to. Would you care to see it?”
“It would be a great pleasure.”
“Please follow me.” He led me through the doorway in the rear of his shop. The next room was a workshop, arrayed with devices whose functions I could not guess—bars of metal wrapped with enough copper thread to reach the horizon, mirrors mounted on a circular slab of granite floating in quick-silver—but Bashaarat walked past these without a glance.
Instead he led me to a sturdy pedestal, chest high, on which a stout metal hoop was mounted upright. The hoop’s opening was as wide as two outstretched hands, and its rim so thick that it would tax the strongest man to carry. The metal was black as night, but polished to such smoothness that, had it been a different color, it could have served as a mirror. Bashaarat bade me stand so that I looked upon the hoop edgewise, while he stood next to its opening.
“Please observe,” he said.
Bashaarat thrust his arm through the hoop from the right side, but it did not extend out from the left. Instead, it was as if his arm were severed at the elbow, and he waved the stump up and down, and then pulled his arm out intact.
I had not expected to see such a learned man perform a conjuror’s trick, but it was well done, and I applauded politely.
“Now wait a moment,” he said as he took a step back.
I waited, and behold, an arm reached out of the hoop from its left side, without a body to hold it up. The sleeve it wore matched Bashaarat’s robe. The arm waved up and down, and then retreated through the hoop until it was gone.
The first trick I had thought a clever mime, but this one seemed far superior, because the pedestal and hoop were clearly too slender to conceal a person. “Very clever!” I exclaimed.
“Thank you, but this is not mere sleight of hand. The right side of the hoop precedes the left by several seconds. To pass through the hoop is to cross that duration instantly.”
“I do not understand,” I said.
“Let me repeat the demonstration.” Again he thrust his arm through the hoop, and his arm disappeared. He smiled, and pulled back and forth as if playing tug-a-rope. Then he pulled his arm out again, and presented his hand to me with the palm open. On it lay a ring I recognized.
“That is my ring!” I checked my hand, and saw that my ring still lay on my finger. “You have conjured up a duplicate.”
“No, this is truly your ring. Wait.”
Again, an arm reached out from the left side. Wishing to discover the mechanism of the trick, I rushed over to grab it by the hand. It was not a false hand, but one fully warm and alive as mine. I pulled on it, and it pulled back. Then, as deft as a pickpocket, the hand slipped the ring from my finger and the arm withdrew into the hoop, vanishing completely.
“My ring is gone!” I exclaimed.
“No, my lord,” he said. “Your ring is here.” And he gave me the ring he held. “Forgive me for my game.”
I replaced it on my finger. “You had the ring before it was taken from me.”
At that moment an arm reached out, this time from the right side of the hoop. “What is this?” I exclaimed. Again I recognized it as his by the sleeve before it withdrew, but I had not seen him reach in.
“Recall,” he said, “the right side of the hoop precedes the left.” And he walked over to the left side of the hoop, and thrust his arm through from that side, and again it disappeared.
Your Majesty has undoubtedly already grasped this, but it was only then that I understood: whatever happened on the right side of the hoop was complemented, a few seconds later, by an event on the left side. “Is this sorcery?” I asked.
“No, my lord, I have never met a djinni, and if I did, I would not trust it to do my bidding. This is a form of alchemy.”
He offered an explanation, speaking of his search for tiny pores in the skin of reality, like the holes that worms bore into wood, and how upon finding one he was able to expand and stretch it the way a glassblower turns a dollop of molten glass into a long-necked pipe, and how he then allowed time to flow like water at one mouth while causing it to thicken like syrup at the other. I confess I did not really understand his words, and cannot testify to their truth. All I could say in response was, “You have created something truly astonishing.”
“Thank you,” he said, “but this is merely a prelude to what I intended to show you.” He bade me follow him into another room, further in the back. There stood a circular doorway whose massive frame was made of the same polished black metal, mounted in the middle of the room.
“What I showed you before was a Gate of Seconds,” he said. “This is a Gate of Years. The two sides of the doorway are separated by a span of twenty years.”
I confess I did not understand his remark immediately. I imagined him reaching his arm in from the right side and waiting twenty years before it emerged from the left side, and it seemed a very obscure magic trick. I said as much, and he laughed. “That is one use for it,” he said, “but consider what would happen if you were to step through.” Standing on the right side, he gestured for me to come closer, and then pointed through the doorway. “Look.”
I looked, and saw that there appeared to be different rugs and pillows on the other side of the room than I had seen when I had entered. I moved my head from side to side, and realized that when I peered through the doorway, I was looking at a different room from the one I stood in.
“You are seeing the room twenty years from now,” said Bashaarat.
I blinked, as one might at an illusion of water in the desert, but what I saw did not change. “And you say I could step through?” I asked.
“You could. And with that step, you would visit the Baghdad of twenty years hence. You could seek out your older self and have a conversation with him. Afterwards, you could step back through the Gate of Years and return to the present day.”
Hearing Bashaarat’s words, I felt as if I were reeling. “You have done this?” I asked him. “You have stepped through?”
“I have, and so have numerous customers of mine.”
“Earlier you said I was the first to whom you showed this.”
“This Gate, yes. But for many years I owned a shop in Cairo, and it was there that I first built a Gate of Years. There were many to whom I showed that Gate, and who made use of it.”
“What did they learn when talking to their older selv
es?”
“Each person learns something different. If you wish, I can tell you the story of one such person.” Bashaarat proceeded to tell me such a story, and if it pleases Your Majesty, I will recount it here.
The Tale of the Fortunate Rope-Maker
There once was a young man named Hassan who was a maker of rope. He stepped through the Gate of Years to see the Cairo of twenty years later, and upon arriving he marveled at how the city had grown. He felt as if he had stepped into a scene embroidered on a tapestry, and even though the city was no more and no less than Cairo, he looked upon the most common sights as objects of wonder.
He was wandering by the Zuweyla Gate, where the sword dancers and snake charmers perform, when an astrologer called to him. “Young man! Do you wish to know the future?”
Hassan laughed. “I know it already,” he said.
“Surely you want to know if wealth awaits you, do you not?”
“I am a rope-maker. I know that it does not.”
“Can you be so sure? What about the renowned merchant Hassan al-Hubbaul, who began as a rope-maker?”
His curiosity aroused, Hassan asked around the market for others who knew of this wealthy merchant, and found that the name was well known. It was said he lived in the wealthy Habbaniya quarter of the city, so Hassan walked there and asked people to point out his house, which turned out to be the largest one on its street.