by Kali Wallace
The last time I had seen Sunita, she had walked me to my quarters aboard Symposium. It was shortly before midnight on a night like any other; we had been in the laboratory late, running Vanguard through a series of tests to simulate what it would do when it lost radio contact with the base on Titan. It had recently developed the strange habit of reaching out to the comms systems of other departments aboard Symposium and persuading their AIs to do its talking for it, and we had to figure out a way to structure our tests so it couldn’t do that. There wouldn’t be anybody else to contact on Titan, after all; it had to learn to rely only on itself and what we brought with us. We had continued our conversation all the way from the lab to our personal berths, speculating on what the overnight results would show, planning a new test, exchanging rapid-fire ideas and adjustments so easily, so comfortably, as we had done for years. I had not known it would be our final conversation. I had not known to say goodbye. I had said, “I’m betting it figures it out,” and Sunita had smiled at me, a beautiful wide smile that lit her entire face, and she had said, “You know our naughty little child best. Sleep well.”
Three hours later I woke to screaming alarms and fire and pain.
I never saw Sunita again. She was gone. Vanguard was gone. Our mission was gone. And I was here, in this hateful box of metal on an ugly rock in the belt, with this cold, smirking woman before me, and I did not know what to say.
“I have so many things I want to ask you about it,” Ping said. “Do you mind? It’s the evolutionary aspect that interests me the most. I can’t say I keep up with the literature as much as I should, but how did you avoid Baldwin’s Law? You must have had precautions in place, to achieve project approval within the disarmament treaty. I know how very particular they are about avoiding the mistakes of the past.”
I glanced at Adisa, but his expression remained blandly unconcerned, as though Ping wasn’t using polite euphemisms to talk about the attempted genocide of his people and destruction of his home. Mistakes of the past was what people said when they wanted to talk about the horrors of the Martian war without acknowledging that those horrors had been entirely intentional. Vice Admiral Dane Baldwin had been responsible for developing and deploying the United Earth Navy’s autonomous weapons on Mars. The threshers that razed the agricultural domes and kick-started a famine, the dusters that destroyed the solar panels and cast entire cities into a deadly winter, the slugs that poisoned the water supply and rendered nearly half of the survivors sterile.
Unintended consequences of technological advancement, Baldwin had said at the tribunal following the war. Not his fault. The machines made their own decisions. The machines were responsible.
I put my PD down and sat forward in my chair. I didn’t know what Mary Ping wanted from me, only that she meant to provoke, but I had been provoked by better than her a hundred times before.
I said, “There’s no such thing as Baldwin’s Law. It has no basis in theory or practice. Artificial intelligences are not inherently destructive. There have been fully evolutionary AIs since Zhao’s first Taijin mind, and most of them don’t turn into killing machines. Baldwin didn’t want to be held accountable for what he had created, so he blamed the machines for his own choices. He was never trying to do anything but create weapons of war.”
“But his excuse convinced the tribunal,” Ping said. “Oh, I know they found him guilty of some minor war crimes, but his only punishment was a few years of house arrest. He’s a free man now.”
That much was true. He had been the invited guest at an AI conference a few years before I left Earth. I had seen him in the hallways of the convention center, a red-faced man in a tailored suit—no sign of his naval uniform—talking in a booming voice while acolytes and admirers scurried behind him, asking questions he never deigned to hear.
“The war tribunal was not made up of experts in artificial intelligence,” I said.
“You truly don’t think violence is inevitable in the evolution of an advanced AI?” Ping asked.
“I know it isn’t.”
“But violence is inevitable in nature,” she said, “and isn’t the goal of an evolving AI to mimic nature as closely as possible? And do we truly understand what happens on the frontier between technology and nature—such as in your own lovely body?”
I curled my left hand into a fist but did not move it from the table.
“Nothing is inevitable with AI,” I said. I would have really liked for Adisa to jump in and get the conversation back on track, but he kept quiet. “The goal of an evolving AI is to improve itself for the tasks it is given, and to do so in ways that we can’t conceive or define. If that task is not a violent one, there is absolutely no reason for the AI to seek violent solutions.”
“Vanguard never did?”
“Vanguard was an explorer,” I said tightly. “Everything it did was toward the goal of collecting as much information as possible in an unfamiliar environment while not disturbing or altering that environment any more than absolutely necessary. Destructive actions would have made that goal harder to achieve.”
“You must have been so very proud of it.”
“Yes. I was.”
“Have you considered creating it again? If it could be done once, surely it could be done twice.”
She wasn’t the first to ask. She wasn’t the first to fail to see how it was such an empty question. If she knew anything at all about artificial intelligences, she had to know no individual AI could ever be exactly replicated. No explorer I might create in the future would choose an elegant praying mantis as its favorite physical form and accept the nickname Bug with a bob of its head and wave of its arms. I would never again design an AI that would learn to play Sunita’s favorite piano concertos when we worked late in the lab, or collect evidence of a grad student stealing engineering tools before we even noticed anything was missing, or organize the data it collected according to which sets it thought would excite me the most. A new AI would never learn to communicate in every possible physical form, from a sturdy six-wheeled rover to a long-winged drone, using an ever-expanding array of elaborate gestures that almost resembled dance and conveyed more nuance than more ordinary stilted natural voice algorithms ever could. A new AI would not teach itself to assign silly food names to each of the team members and start using them without warning, causing us to be baffled, then delighted, as we tried to work out who was Gyoza or Pickles or Baba Ghanoush. It wouldn’t learn to taunt less complex robots with its agility or make up logic games to challenge other AIs. All of the things Vanguard had learned and discovered while we were making it the smartest explorer it could be, they were gone and could never be replicated.
My voice was hollow when I answered. “It wouldn’t be the same.”
“No, no, of course not. That is rather like asking a mother to replace a dead child with a younger sibling, I suppose. The joy is in the unpredictability, isn’t it? You must have given it so much freedom, to let it grow so powerful. I worry that since the Aeolia incident we’re neutering our Overseers by restricting them too much.”
I leaned back, watching Ping carefully. “Do you think Nimue’s Overseer is at risk of an attack like Aeolia?”
“Oh, no,” she said quickly. “I’ve seen no sign of that. But it’s something David and I talked about, although he didn’t share my concerns.”
“What exactly did you talk about?” I asked. I glanced at Adisa, a clear invitation to jump in. I didn’t know the first fucking thing about Aeolia except that it kept coming up in this investigation and everybody knew more about it than I did. He remained stubbornly, uselessly silent. “I mean with regards to Aeolia.”
“Only what was relevant for our jobs. The changes implemented after the incident made our jobs a bit duller, to be honest. It might be good for business—and for safety, I suppose—but I’m not sure it’s good for them. For the Overseers.”
I couldn’t st
op myself from making a face at that. “They’re machines.”
“I’m surprised to hear you of all people say that,” Ping said. Her accent was slipping, letting a bit of rough-and-tumble orbital rat shine through. “With what you created. And what you are now.”
There it was again. The reason for her staring, her questions, her focus.
“I don’t understand. What am I now?” I asked. “The untamed frontier between nature and technology?”
She dipped her chin slightly. “I’ve offended you.”
“Do you really believe in that?”
“You don’t?” she said. “Wasn’t it the Zhao herself who said that when our machines know us as well as we think we know them, the distinction will be irrelevant?”
“She also said she would wager her life’s savings on a Yuèliàng kite-jack race before she would try to predict the future of AI,” I pointed out. “Yet people keep trying to predict the future of AI. As they’ve been trying and failing to do for centuries.”
“Ah, well, she had her quirks, our mother of machines,” Ping said. She sat forward in her chair and extended one hand toward me. “I’m sorry. I know this is inappropriate, but I can’t help myself. May I look at your arm?”
I didn’t move. Not so much as a twitch of the fingers. “No,” I said.
“I’ve made you uncomfortable.” Ping sat back and withdrew her hand, curling her fingers closed as she did so. “I didn’t mean to. It’s not mere prurient interest. My curiosity is professional. Who did the work? I only want to look. And, if I may”—a self-conscious laugh—“touch, just a bit? You must tell me who did your work. It’s stunning.”
I was surprised that she came right out and said it, as though there was nothing inappropriate about the hungry look in her eyes, the way she reached before she asked. Would she have said the same to the boy with the bleeding eyes, I wondered, and envied him for how his brain and body had been butchered? I had met the man who designed my prosthetic limbs only a few times in the hospital on Badenia, between my many surgeries. He had called me “Helen” and “girl” and “people like you” and asked me repeatedly if I was sure I didn’t want my new arm and leg to match my skin, he had a lovely golden tone they could use, it would match perfectly if I spent some extra time under UV light, and it would only cost a little more, I should really consider it, it was sure to be all the rage among his female patients who wished to remain beautiful while redefining humanity. It was a relief when he left me to the surgeons and nurses, to his uncaring legal representatives and bored liaisons. They all wanted me to be very clear on what would happen if I should leave Parthenope before my medical bills were paid in full (repossession), if I should allow a third party to study the prosthetics in such a manner as to encourage unauthorized reproduction (prosecution), if I should make public statements disparaging the doctor and/or his team associated with Parthenope Medical and/or the technology of which I was currently in possession (litigation), if I should alter and/or modify and/or damage the patented and proprietary Augmented Medical Devices in any way (all of the above). I signed everything. I had no choice. I declined every one of their entreaties to offer myself up to the laboratory for further research or promotional duties.
I had chosen naked metal instead of the nauseating facsimile of human skin.
I stared at Mary Ping. I wanted to say no again, but I was afraid I would shout, or vomit, or cry. I kept my hand on the table.
“Is that all you want to talk about?” I said. “Because I don’t have time for a philosophical debate about the existential evolution of artificial intelligences. Do you know anything else about David’s death? Anything you haven’t told us?”
She didn’t flinch. She just kept staring at me, looking me right in my prosthetic eye. “I do not. But please do let me know what I can do to help,” she said. For the first time in what felt like an eternity, she turned from me to look at Adisa. “I know it must be hard for you to investigate under these circumstances. You normally don’t do much but watch the surveillance and drag the scapegoat away.”
“We sometimes do a bit more than that, aye,” Adisa said.
“Of course.” She rose from her chair but hesitated before stepping away from the table. “If that’s all, I have work to do. Twice as much, now. I’m sure you understand.”
“Wait,” I said. I had almost forgotten. “Do you know of any Parthenope project or operation with a name like Sunshine or Sunlight?”
“No, but I’m hardly aware of much outside my area of expertise,” Ping said. “I really do have to get to—”
“One more question,” said Adisa.
Ping barely hid her annoyance. “Yes?”
“Have you ever had reason to suspect there is any criminal activity happening on Nimue?”
“Really, Safety Inspector, I would have reported it, if I did.”
“Not even a hunch?”
“No. Nothing.” She took a step, turned back to look at me. “Are they all in prison now? The people responsible for Symposium, those who survived. I saw that it was in the news. I even asked David about it, but he never liked to talk about it. I do wish he had reached out for comfort when he needed it, before the end.”
My left hand clenched, metal fingers scraping over the tabletop. She knew. I didn’t know how it was possible. I had no proof. I didn’t have any reason for my certainty beyond the tight pain in my chest. But I was absolutely sure she knew David had sent a message to me before he died.
I had assumed his remark about somebody listening had been general, for whoever in the company might be listening, but I knew better now. Mary Ping was the one he had been hiding from with his cryptic memories and awkward code talk.
As soon as she was gone, I wanted to call her back, take her by the shirt and shake her, ask her what she knew and what she had done. Why she had asked me about Vanguard, about violent AIs, about the evolution of machines. What David had told her. Why any of it mattered.
I found myself thinking about the time I had taken Vanguard to the bottom of the ocean and what it had done there. It had been a few years before Symposium launched. I hadn’t thought of those days in ages, but now they were filling my mind again, dancing around with Mary Ping’s sly question: Vanguard never did?
The rivalry we had with the members of the Europa Deep-Sea Expedition was more antagonistic than friendly. In one respect, they were years ahead of the Titan project on every possible axis. There were already colonies on Europa with transportation connections that kept the bases supplied with both people and resources. They had been drilling into the ice for two years already. They would be sending their autonomous submersibles into that cold, dark ocean well before we landed on Titan. They were probably going to find life first. We could all admit that—if not to their faces, but to each other, after a few drinks—and it stung.
But because our goals were so similar, if our destinations so different, we often found ourselves working side by side. That’s what happened when I took a portable detachment of Vanguard down to the Joint Territories Mid-Atlantic Research Station, which sat on the bottom of the ocean near the thermal vent colonies on the Mid-Atlantic Ridge. My goal at Mid-AR was to train Vanguard in a wholly unusual and unexpected environment. I knew it could adapt to exploring underwater and at high pressure, but I had never taught it anything about the living colonies of organisms around the deep-sea vents. I wanted to see what it would do when it encountered those bizarre creatures for the first time, those life-forms that didn’t play by the rules of energy acquisition and resource management that we had introduced it to on the surface.
The first few trial runs did not go well. Vanguard didn’t like the high pressure at that depth, and it had balked at exploring far from the station when I sent it out to get its sea legs. That failure, unfortunately, happened right when one of the submersible designers from the Europa expedition was watching.
&
nbsp; “You’re asking too much of it,” the insufferable Rodney Grieg said, when Vanguard curled itself into a ball.
“You’ve got to give it a little programming push,” said Grieg, when Vanguard kept itself close to the station.
“You need to look for errors in your algorithm,” was Grieg’s advice when I brought Vanguard back inside.
“You need to kill that bug shape,” declared Grieg, with a bit of a shudder, when Vanguard re-formed into its praying mantis form to scurry back to the laboratory.
“These kinds of machines, they aren’t easy to design,” Grieg said, over dinner one night, pointing his fork at me as he spoke. “There are a lot of special considerations. I can go over them with you later. I’ve got a free hour.”
It was all I could do not to snatch the fork from his hand and stab it into his eye.
Instead I slunk away to my berth to spend the rest of the evening alone. The Europa team were all as obnoxious as Grieg, and the crew of Mid-AR were an insular and wary bunch. I worried when I interacted with them that I was looking at my future, that years in the darkness and isolation of space would teach me to stare too long, to linger by dark portholes, to treat strangers with open suspicion. I preferred Vanguard’s company. It crawled around my berth as Bug, its praying mantis form, exploring every nook and corner of the little room, while I poked at the programming of its test parameters. It kept interrupting me to share nuggets of information about Grieg’s team: who had plagiarized their thesis, who had finessed test results, who was accepting small bribes from corporate entities to share bits of research. I didn’t really care about the ethically questionable choices of the Europa team, but Vanguard was a shameless gossip and I never felt lonely when it was sharing its findings with me.
I had still not solved the problem by the next morning, when it was once again time to send Vanguard out for a test. Grieg was loudly complaining about a slow pump in the hatch, so neither he nor anybody else noticed when I gave Vanguard a little pat on its triangle head—it had been years since I’d felt ridiculous for such habits, as Vanguard spoke most eloquently in gestures—and whispered, “Go out there and make me proud, kid.”