by Ted Chiang
Ellen’s face was unreadable for a moment.
“I think I should head home,” she said. “Early workday tomorrow.” She looked away.
“Did Tilly tell you to say that?”
She said nothing and avoided looking into his eyes.
“I had a great time,” Sai added quickly. “Would you like to go out again?”
Ellen paid half the bill and did not ask him to walk her home.
“You’re being very antisocial tonight,” Tilly said.
“I’m not antisocial. I just didn’t like how you were interfering with everything.”
“I have every confidence you would have enjoyed the rest of the date had you followed my advice.”
Sai drove on in silence.
“I sense a lot of aggression in you. How about some kickboxing? You haven’t gone in a while, and there’s a 24-hour gym coming up. Take a right here.”
Sai drove straight on.
“What’s wrong?”
“I don’t feel like spending more money.”
“You know I have a coupon.”
“What exactly do you have against me saving my money?”
“Your savings rate is right on target. I simply want to make sure you’re sticking to your regimen for consumption of leisure. If you over-save, you’ll later regret that you didn’t make the most of your youth. I’ve plotted the optimum amount of consumption you should engage in daily.”
“Tilly, I just want to go home and sleep. Can you shut yourself off for the rest of the night?”
“You know that in order to make the best life recommendations, I need to have complete knowledge of you. If you shut me out of parts of your life, my recommendations won’t be as accurate—”
Sai reached into his pocket and turned off the phone. The earpiece went silent.
When Sai got home, he saw that the light over the stairs leading up to his apartment had gone out, and several dark shapes skulked around the bottom.
“Who’s there?”
Several of the shadows scattered, but one came toward him: Jenny.
“You’re back early.”
He almost didn’t recognize her; this was the first time he’d heard her voice without the electronic filter she normally used. It sounded surprisingly … happy.
Sai was taken aback. “How did you know I was back early? You stalking me?”
Jenny rolled her eyes. “Why would I need to stalk you? Your phone automatically checks in and out of everywhere you go with a status message based on your mood. It’s all on your ShareAll lifecast for anyone to see.”
He stared at her. In the faint glow from the streetlights he could see that she wasn’t wearing her thick winter coat or ski goggles or scarf. Instead, she was in shorts and a loose white t-shirt. Her black hair had been dyed white in streaks. In fact, she looked very pretty, if a bit nerdy.
“What, surprised that I do know how to use a computer?”
“It’s just that you usually seem so …”
“Paranoid? Crazy? Say what’s on your mind. I won’t be offended.”
“Where’s your coat and goggles? I’ve never even seen you without them.”
“Oh, I taped over your door camera so my friends could come for a visit tonight, so I’m not wearing them. I’m sorry—”
“You did what?”
“—and I came out here to meet you because I saw that you turned off Tilly, not once, but twice. I’m guessing you’re finally ready for the truth.”
Stepping into Jenny’s apartment was like stepping into the middle of a fishing net.
The ceiling, floor, and walls were all covered with a fine metal mesh, which glinted like liquid silver in the flickering light from the many large, hi-definition computer monitors stacked on top of each other around the room, apparently the only sources of illumination.
Besides the monitors, the only other visible furniture appeared to be bookshelves—full of books (the paper kind, strangely enough). A few upside-down, ancient milk crates covered with cushions served as chairs.
Sai had been feeling restless, had wanted to do something strange. But he now regretted his decision to accept her invitation to come in. She was indeed eccentric, perhaps too much so.
Jenny closed the door and reached up and plucked the earpiece out of Sai’s ear. Then she held out her hand. “Give me your phone.”
“Why? It’s already off.”
Jenny’s hand didn’t move. Reluctantly, Sai took out his phone and gave it to her.
She looked at it contemptuously. “No removable battery. Just what you’d expect of a Centillion phone. They should call these things tracking devices, not phones. You can never be sure they’re really off.” She slipped the phone inside a thick pouch, sealed it, and dropped it on the desk.
“Okay, now that your phone is acoustically and electromagnetically shielded, we can talk. The mesh on the walls basically makes my apartment into a Faraday cage, so cellular signals can’t get through. But I don’t feel comfortable around a Centillion phone until I can put a few layers of shielding around it.”
“I’m just going to say it. You are nuts. You think Centillion spies on you? Their privacy policy is the best in the business. Every bit of information they gather has to be given up by the user voluntarily, and it’s all used to make the user’s life better—”
Jenny tilted her head and looked at him with a smirk until he stopped talking.
“If that’s all true, why did you turn Tilly off tonight? Why did you agree to come up here with me?”
Sai wasn’t sure he himself knew the answers.
“Look at you. You’ve agreed to have cameras observe your every move, to have every thought, word, interaction recorded in some distant data center so that algorithms could be run over them, mining them for data that marketers pay for.
“Now you’ve got nothing left that’s private, nothing that’s yours and yours alone. Centillion owns all of you. You don’t even know who you are any more. You buy what Centillion wants you to buy; you read what Centillion suggests you read; you date who Centillion thinks you should date. But are you really happy?”
“That’s an outdated way to look at it. Everything Tilly suggests to me has been scientifically proven to fit my taste profile, to be something I’d like.”
“You mean some advertiser paid Centillion to pitch it at you.”
“That’s the point of advertising, isn’t it? To match desire with satisfaction. There are thousands of products in this world that would have been perfect for me, but I might never have known about them. Just like there’s a perfect girl out there for me, but I might never have met her. What’s wrong with listening to Tilly so that the perfect product finds the perfect consumer, the perfect girl finds the perfect boy?”
Jenny chuckled. “I love how you’re so good at rationalizing your state. I ask you again: If life with Tilly is so wonderful, why did you turn her off tonight?”
“I can’t explain it,” Sai said. He shook his head. “This is a mistake. I think I’ll head home.”
“Wait. Let me show you a few things about your beloved Tilly first,” Jenny said. She went to the desk and started typing, bringing up a series of documents on a monitor. She talked as Sai tried to scan them and get their gist.
“Years ago, they caught Centillion’s traffic-monitoring cars sniffing all the wireless traffic from home networks on the streets they drove through. Centillion also used to override the security settings on your machine and track your browsing habits before they shifted to an opt-in monitoring policy designed to provide better ‘recommendations.’ Do you think they’ve really changed? They hunger for data about you—the more the better—and damned if they care about how they get it.”
Sai flicked through the documents skeptically. “If this is all true, why hasn’t anyone brought it up in the news?”
Jenny laughed. “First, everything Centillion did was arguably legal. The wireless transmissions were floating in public space, for example, so there
was no violation of privacy. And the end user agreement could be read to allow everything Centillion did to ‘make things better’ for you. Second, these days, how do you get your news except through Centillion? If Centillion doesn’t want you to see something, you won’t.”
“So how did you find these documents?”
“My machine is connected to a network built on top of the Net, one that Centillion can’t see inside. Basically, we rely on a virus that turns people’s computers into relaying stations for us, and everything is encrypted and bounced around so that Centillion can’t see our traffic.”
Sai shook his head. “You’re really one of those tinfoil-hat conspiracy theorists. You make Centillion sound like some evil repressive government. But it’s just a company trying to make some money.”
Jenny shook her head. “Surveillance is surveillance. I can never understand why some people think it matters whether it’s the government doing it to you or a company. These days, Centillion is bigger than governments. Remember, it managed to topple three countries’ governments just because they dared to ban Centillion within their borders.”
“Those were repressive places—”
“Oh, right, and you live in the land of the free. You think Centillion was trying to promote freedom? They wanted to be able to get in there and monitor everyone and urge them to all consume more so that Centillion could make more money.”
“But that’s just business. It’s not the same thing as evil.”
“You say that, but that’s only because you don’t know what the world really looks like any more, now that it’s been remade in Centillion’s image.”
Although Jenny’s car was heavily shielded like her apartment, as she and Sai drove, she whispered anyway, as if she were afraid that their conversation would be overheard by people walking by on the sidewalk.
“I can’t believe how decrepit this place looks,” Sai said as she parked the car by the side of the street. The surface of the road was pockmarked with potholes and the houses around them in ill repair. A few had been abandoned and were falling apart. In the distance they could hear the fading sound of a police siren. This was not a part of Las Aldamas that Sai had ever been to.
“It wasn’t like this even ten years ago.”
“What happened?”
“Centillion noticed a certain tendency for people—some people, not all—to self-segregate by race when it came to where they wanted to live. The company tried to serve this need by prioritizing different real estate listings to searchers based on their race. Nothing illegal about what they were doing, since they were just satisfying a need and desire in their users. They weren’t hiding any listings, just pushing them far down the list, and in any event, you couldn’t ever pick apart their algorithm and prove that they were looking at race when it was just one out of hundreds of factors in their magical ranking formula.
“After a while, the process began to snowball, and the segregation got worse and worse. It became easier for the politicians to gerrymander districts based on race. And so here we are. Guess who got stuck in these parts of the town?”
Sai took a deep breath. “I had no idea.”
“If you ask Centillion, they’ll say that their algorithms just reflected and replicated the desire to self-segregate in some of their users, and that Centillion wasn’t in the business of policing thoughts. Oh, they’d claim that they were actually increasing freedom by giving people just what they wanted. They’d neglect to mention that they were profiting off of it through real estate commissions, of course.”
“I can’t believe no one ever says anything about this.”
“You’re forgetting again that everything you know now comes filtered through Centillion. Whenever you do a search, whenever you hear a news digest, it’s been curated by Centillion to fit what it thinks you want to hear. Someone upset by the news isn’t going to buy anything sold by the advertisers, so Centillion adjusts things to make it all okay.
“It’s like we’re all living in Oz’s Emerald City. Centillion puts these thick green goggles over our eyes and we all think everything is a beautiful shade of green.”
“You’re accusing Centillion of censorship.”
“No. Centillion is an algorithm that’s gotten out of hand. It just gives you more of what it thinks you want. And we—people like me—think that’s the root of the problem. Centillion has put us in little bubbles, where all we see and hear are echoes of ourselves, and we become ever more stuck in our existing beliefs and exaggerated in our inclinations. We stop asking questions and accept Tilly’s judgment on everything.
“Year after year, we become more docile and grow more wool for Centillion to shave off and grow rich with. But I don’t want to live that way.”
“And why are you telling me all this?”
“Because, neighbor, we’re going to kill Tilly,” Jenny said, giving Sai a hard look, “and you’re going to help us do it.”
Jenny’s apartment, with all its windows tightly shut and curtained, felt even more stifling after the car ride. Sai looked around at the flickering screens showing dancing, abstract patterns, suddenly wary. “And just how are you planning to kill Tilly, exactly?”
“We’re working on a virus, a cyber weapon, if you want to get all macho about it.”
“What exactly would it do?”
“Since the lifeblood of Tilly is data—the billions of profiles Centillion has compiled on every user—that’s how we have to take it down.
“Once inside the Centillion data center, the virus will gradually alter every user profile it encounters and create new, fake profiles. We want it to move slowly to avoid detection. But eventually, it will have poisoned the data so much that it will no longer be possible for Tilly to make creepy, controlling predictions about users. And if we do it slowly enough, they can’t even go to backups because they’ll be corrupted too. Without the data it’s built up over the decades, Centillion’s advertising revenue will dry up overnight, and poof, Tilly’ll be gone.”
Sai imagined the billions of bits in the cloud: his tastes, likes and dislikes, secret desires, announced intentions, history of searches, purchases, articles and books read, pages browsed.
Collectively, the bits made up a digital copy of him, literally. Was there anything that was a part of him that wasn’t also up there in the cloud, curated by Tilly? Wouldn’t unleashing a virus on that be like suicide, like murder?
But then he remembered how it had felt to have Tilly lead him by the nose on every choice, how he had been content, like a pig happily wallowing in his enclosure.
The bits were his, but not him. He had a will that could not be captured in bits. And Tilly had almost succeeded in making him forget that.
“How can I help?” Sai asked.
Sai woke to Miles Davis’s rendition of “So What.”
For a moment, he wondered if the memory of the night before wasn’t a dream. It felt so good to be awake, listening to just the song he wanted to hear.
“Are you feeling better, Sai?” Tilly asked.
Am I?
“I thought I turned you off, Tilly, with a hardware switch.”
“I was quite concerned that you stopped all Centillion access to your life last night and forgot to turn it back on. You might have missed your wake-up call. However, Centillion added a system-level fail-safe to prevent just such an occurrence. We thought most users such as yourself would want such an override so that Centillion could regain access to your life.”
“Of course,” Sai said. So it’s impossible to turn Tilly off and keep her off. Everything Jenny said last night was true. He felt a chill tingle on his back.
“There’s a gap of about twelve hours during which I couldn’t acquire data about you. To prevent degradation in my ability to help you, I recommend that you fill me in.”
“Oh, you didn’t miss much. I came home and fell asleep. Too tired.”
“There appears to have been vandalism last night of the new security cameras you
installed. The police have been informed. Unfortunately, the camera did not capture a good image of the perpetrator.”
“Don’t worry about it. There’s nothing here worth stealing anyway.”
“You sound a bit down. Is it because of the date last night? It seems that Ellen wasn’t the right match for you after all.”
“Um, yeah. Maybe not.”
“Don’t worry: I know just the thing that will put you in a good mood.”
Over the next few weeks, Sai found it extremely difficult to play his assigned role.
Maintaining the pretense that he still trusted Tilly was crucial, Jenny had emphasized, if their plan was to succeed. Tilly couldn’t suspect anything was going on at all.
It seemed simple enough at first, but it was nerve wracking, keeping secrets from Tilly. Could she detect the tremors in his voice, Sai wondered. Could she tell that he was faking enthusiasm for the commercial consumption transactions she suggested?
Meanwhile, he also had a much bigger puzzle to solve before John P. Rushgore, Assistant General Counsel of Centillion, came to Chapman Singh in another week.
Chapman Singh is defending Centillion in a patent dispute with ShareAll, Jenny had said. This is our opportunity to get inside Centillion’s network. All you have to do is to get someone from Centillion to plug this into his laptop.
And she had then handed him a tiny thumb drive.
Though he still hadn’t figured out a plan for plugging the thumb drive into a Centillion machine, Sai was glad to have come to the end of another long day of guarding himself against Tilly.
“Tilly, I’m going jogging. I’ll leave you here.”
“You know that it’s best to carry me with you,” Tilly said. “I can track your heart rate and suggest an optimal route for you.”
“I know. But I just want to run around on my own a bit, all right?”
“I’m growing quite concerned with your latest tendencies towards hiding instead of sharing.”
“There’s no tendency, Tilly. I just don’t want you to be stolen if I get mugged. You know this neighborhood has become more unsafe lately.”
And he turned off the phone and left it in his bedroom.