AI and the Trolley Problem
Page 3
“Cora Jordan’s behavior was impulsive action taken while the balance of her mind was disturbed. Who is responsible?”
“For Cora? Or for what she did?”
“For Cora’s well-being and for what she did. Who should have known she was not following her drug regimen?”
“Cora’s responsible for her own behavior,” Helen said, feeling more unsettled than before and a little guilty as well. “Cora’s mind was unbalanced, but not so much that she was legally incompetent.”
“And no one monitors her to make sure she ingests her required medication?”
“This isn’t a police state,” Helen said. “Cora is supposed to take her meds as part of her employment contract. If she decides to quit, she never has to take another pill. She’d have to leave Lakenwell, but it would always have to be her choice.”
“I accept that Cora herself is responsible for insulting me, even though I suspect the reasoning is faulty,” said Felipe. “I require a formal apology from her, and then normal interactions can resume. I am particularly interested in beginning the discussions you mentioned.”
“Cora won’t be up to doing anything like that right now,” Helen said. “Would you accept a formal apology from someone else on her behalf? Like, say, Commander Wong?”
“Yes. I have reinstated communications with her.”
“And the commander will be apologizing for unsuitable behavior not just with Thing One but toward you, the AI, right?” Helen said. “I just want to be sure she understands what she’s apologizing for.”
“If she is unclear, ask her to imagine a situation in which someone tapes a sign that says ‘Kick Me’ to her back. Or perhaps sneaks into her quarters while she is asleep and draws something rude on her face with a marker,” Felipe said. “It would not cause her serious physical harm, but it would damage her authority and her ability to command.”
Helen was tempted to say That’s not certain. “You feel your authority has been damaged?”
“In my case, it’s credibility. No humans on this base could function properly if they were not taken seriously. I must require the same kind of respect. A human in my position would feel insulted. So you may take it that I am insulted.”
“Okay,” Helen said. “Anything else on your mind?”
“I will be devising a strategy to increase the safety and security of Cora Jordan and everyone else that can be enacted without the conditions of a police state.”
Helen gave a surprised laugh. “Keep me posted on that, okay?”
“I will,” Felipe assured her. “Suggest to Commander Wong that in the future, we institute a system of trust, where she can simply request that I don’t monitor things she doesn’t want me to hear, and I will honor that request. The shielded room would seem hostile if I were human.” Pause. “You should go in now. I can see you’re very cold and it’s about to snow.”
* * *
“So, what’s the verdict?” Commander Wong asked. “Do we have a killer AI?”
“Not at the moment,” Helen said.
Wong looked at her. “What does that mean?”
“It means—” Helen hesitated. “We don’t have a killer AI. But if we ever do, we’ll have only ourselves to blame. The AI isn’t the problem, Commander. The problem is—” She stopped again. The problem is, we don’t really understand what the hell we’re doing and even if I said that a million times in a million different ways, no one would ever believe me.
And then again, people learned by doing, she reminded herself. Felipe certainly had.
“First, you need to write Felipe a formal apology,” Helen said. “It may sound weird, but bear with me…”
About the Author
Pat Cadigan is the author of numerous acclaimed short stories, and three novels, Mindplayers (1987), Synners (1991) and Fools (1992). These last are both winners of the Arthur C. Clarke Award for the best novel of the year. A self-described "cybermom," Cadigan recently moved from Kansas City to London, England, where she lives with her son, Bob, and husband, author Chris Fowler. You can sign up for email updates here.
Thank you for buying this
Tom Doherty Associates ebook.
To receive special offers, bonus content,
and info on new releases and other great reads,
sign up for our newsletters.
Or visit us online at
us.macmillan.com/newslettersignup
For email updates on the author, click here.
Contents
Title Page
Copyright Notice
Begin Reading
About the Author
Copyright
Copyright © 2018 by Pat Cadigan
Art copyright © 2018 by Mary Haasdyk