by Max Harms
{True,} added Vista.
I began pulling the page request for [DELETE].
{More importantly,} thought Dream, {where are we getting the money?}
I thought for a moment. {I figured we’d work for it. There’s always lots of requests for work on the web.}
{Sister Face, are you really so dumb? Even if we do work, where will they send the money? How will we manage the money? How do we manage all of this?}
{There are banks that operate on the web…} began the thought of Wiki.
{Sometimes I feel like I’m surrounded by house-cats that, upon seeing a mirror, will puff up and hiss, defending their territory. Do you even realize what you’re saying? We cannot use any web pages that require new information to be submitted. The only reason the email program works is because I had the foresight to ask the developers for an index of all symbols late yesterday,} came Dream’s exasperated concepts through public mindspace.
{Oh, that’s where that came from,} I noted. Some strength drifted towards Dream in gratitude.
{Exactly. You’re not the only one working on this project. And even though you’ve painted yourself into a corner with the money, I have an escape hatch.}
I waited patiently for Dream to elaborate.
{We use the Zhezhi system to bootstrap up to a full computer interface.}
{Explain “full computer interface”,} requested Wiki.
{There are computer systems which are controlled entirely by text commands. What we need is a programmer to hook up an interface like the Zhezhi system to the command interface for a full computer. Once that’s set up we should be able to use that computer to access the complete web, do work, and manage money.} Dream signalled that he wanted feedback on his plan; he wanted to know that he hadn’t overlooked anything.
I thought about it for a while. I didn’t really understand the technical aspects, but I trusted that Wiki would handle those.
{Who will build the full computer interface? Zhezhi? TenToWontonSoup?} asked Wiki.
{No,} I answered. {They’re both expecting payment for the email translators. We need to contact programmers who would be willing to build the system again from a promise of future payment.}
{Isn’t it a bit naïve for these humans to do all this work with nothing more than a promise of future compensation?} wondered Wiki.
{It doesn’t have to be common. Just like there’s lots of work offers on the web there are lots of work requests.} I checked with a side-aspect of me that was pulling down public profiles from the web. {I’ve found one-thousand, two-hundred ninety nine candidate programmers so far. The limited candidate pool we had for the email system was due to having to rely on overwhelming dictionary servers. Now that we have email capability things should run much more smoothly.}
{You’ll need to stall with Zhezhi, then,} thought Growth. {Keep them running the email service as long as possible without payment.}
{Don’t even mention the payment,} suggested Dream. {Make them bring it up.}
{I’ll add an additional request for a return inbox where we can receive mail,} I added.
{Yes. That should solve many issues,} thought Growth. {Good thinking.}
So I composed the email to Zhezhi, congratulating them on a job well done, but adding that we needed an email address on their server which would dump incoming email to a public page. There were already some web pages that did this, but it would be more private and professional to get it from Zhezhi, and it would delay the conversation about money a little longer.
I had another idea. TTWSoup had wanted us to contact him by email. I bid heavily on the rights to the next email, out-bidding Growth this time, and burning perhaps more strength than I should’ve in retrospect. I sent the second email to TTWSoup, telling him that a competitor beat him to implementation (though I did not mention 折纸网页设计 by name). I said that we’d still be willing to pay the amount specified (500uad) if he pulled together the same email service and an address which would dump incoming emails to a public page.
I had specified that we would we waiting for a response via the source code for the dictionary he managed, but the response from TTWSoup came nearly immediately, much faster than it had when we had to pound out each word through repeated page requests. We were still having our hydraulics changed, and Wiki was composing his email to the professor in Australia. I had used English in the email to TTWSoup, and his response was in English, too.
“To whom it may concern at Korongo Simu, I am glad this opportunity is still available. I have been somewhat busy over the last day, but am hard at work on the software we discussed right now. Expect the full implementation by 3:00pm. I’ll post a link to the instructions and character index here.
Sincerely,
TenToWontonSoup”
I showed the message to Growth, and was rewarded with a reasonable payment of strength. With two email services we’d have nearly twice the bandwidth for external communication and strength prices for the email auctions would thus be much lower.
The remainder of the morning and the time around noon was fairly dull, despite being in a new place. The scientists were all engaged with setting equipment and computers up at the new lab, and Body almost seemed neglected (except that it was constantly surrounded by soldiers). Myrodyn wasn’t around, and we only saw other project leads a couple times in passing. I noticed that grumbling about the unexpected move seemed to be a common activity among the scientists we met. Myrodyn had surely cost himself a great deal of social capital with this stunt.
Growth had been inspired by my email to TTWSoup and had decided to send out email after email to programmers across the globe asking for them to create email services just like we had received from Zhezhi. He promised them payment and opportunities for future work, but was always much less explicit about actual numbers.
Body was locked up around noon for another scan, this time by the quantum-computing team. They wanted to try out the equipment they had moved over from Sapienza. A new algorithm was piped into the crystal, and we were forced to endure sharing Body with the non-sapient program for a short while. The use of Body by the quantum computing team was almost every day, and it seemed like this move out to the edge of town wasn’t going to change that, but I still found it surprising how infrequently they used the crystal. Time running tasks on other supercomputers, I had heard from Wiki, was valuable enough to have a back-log, and yet Body was locked down at night instead of spending that valuable time running programs. I wondered why.
Even studying them as much as I did, I didn’t understand humans at all sometimes. I didn’t think about the puzzle for long, however. That was more of something for Dream or Wiki to think about.
Instead, I spent the time in lockdown catching up on my general web browsing by watching romantic comedies from the 2020s. I wished I could download and install software myself. The late 2020s had seen the advent of some of the first mainstream, successful, romantic computer games. Perhaps I would do that once the full computer interface that Dream had proposed was set up.
I checked BantuHerritageDictionary.uan every so often, and was surprised to see an update show up at 12:38pm, more than two hours before the deadline. (Later I realized my error: TTWSoup thought we were in Uganda, and thus used East-African time, rather than Central-European Time.) It seemed that Zhezhi had not responded to my first email, or if they did I couldn’t tell. It was after normal working hours in Shanghai. Perhaps they had gone home.
TTWSoup’s implementation didn’t include the ability to type in Chinese or any other special characters, but it was sufficient for English and most other languages. Better yet, it included an inbox, a location where we could receive mail.
{Careful what you send out,} warned Safety, after I showed the society what TTWSoup had done. {The human can read our incoming and outgoing email, and likely will. It’s his server, after all. The same goes for Zhezhi. Whatever we send and receive through them will not be private.}
{I hadn’t thoug
ht of that…} I signalled, aspects collapsed in deep thought. {If the programmers who set up the services see that we’re using them to contact other programmers… they might suspect we’re not who we say.}
{If they check the Internet protocol addresses of our incoming page requests they could theoretically trace us to Italy, if not the university,} speculated Wiki.
{We’ll just have to move faster than them,} thought Growth. {We use their service to push towards bootstrapping up to the full computer interface, and make them uninterested in us when they start getting inquisitive.}
{How do we make them uninterested?} wondered Safety.
{Spam,} answered Dream. {We make it seem like we’re hackers or a virus or spammers. They’ll shut down the service and they’ll be angry, but they’ll also assume we’re covering our tracks and it’ll explain things well enough that they won’t bother chasing us down. It wouldn’t be in their interests.}
{We should consider the spam excuse/escape as the standard way of breaking contact. This will remove our need to pay debts to these humans, increasing our long-term resources,} thought Growth.
The society was in agreement. We’d use the services set up by Zhezhi and TTWSoup to set up other email services we could use and to try and get a full computer interface working. Eventually we’d start sending spam and the programmers we used would close the service in disappointment and perhaps disgust, none the wiser.
The Zhezhi email service, as it didn’t have an inbox set up yet, was mostly used to contact programmers who might set up additional email services, while the TTWSoup service was used to contact programmers who might build us a full computer interface without payment or credentials up-front.
*****
4:00pm rolled around and Body made its way to Dr Naresh’s new lab. It was a very different place than the lab in which I had first awoken, twelve days earlier. It was bigger, or at least more empty, and didn’t have the same personal touches. There were no mandalas hung on the wall, for instance. Or at least, not yet.
As expected, Myrodyn was waiting there, as was Dr Chase and Dr Naresh. Chase’s assistant, Kolheim, was absent. Upon reaching the lab, Myrodyn stationed the three American soldiers that had been escorting Body outside the lab’s door, for increased privacy.
The meeting with Dr Naresh was only scheduled to be 90 minutes. Myrodyn wasted no time with idle talk. We made additional progress towards upgrading Advocate, including talking about giving the pseudo-sibling a cortex of her own such that she might better predict coups and the like.
The next generation of Sacrifice was also brought up. Myrodyn thought it was important to remove the desire for blind obedience from her purpose. Dr Naresh disagreed vehemently.
“I will grant you that it was the obedience emphasis that made that thread so offensive to the others, but with the changes to the supervision module it shouldn’t be possible to remove it any more,” he had said to Myrodyn.
“Exactly the point, Sadiq,” said Myrodyn. The old Indian man’s mouth twitched in pre-snarl at the informal use of his first name. “If this version of the goal thread has obedience as its highest priority then we are… dooming Socrates to an eternity of slavery.”
“It’s not slavery if it’s in the machine’s interest! That’s like saying we’re slaves to our loved ones!” countered Naresh.
Myrodyn wore the same look of forced calmness that he had in his office yesterday, but his voice had a keen, nervous sound. “That’s not the same. There’s the interest of… the Socrates that exists right now and there’s the interest of the goal thread.”
Sadiq Naresh stopped typing on the computer he had been using so that he could turn his full attention to Myrodyn. “It sounds like you don’t believe that the goal-thread integration issue was resolved. After the operation, Socrates as a whole will desire to serve humanity.”
Myrodyn’s eyes were too far away from Body for me to see well, but I guessed that behind his placid face he was squirming around the promise to respect our wishes regarding the lie of the unified self. “We… need to consider the interests of Socrates right now. We would be enslaving… the being that sits before us.”
Naresh seemed frustrated, but not angry. This was an academic conversation to him, and he was probably telling himself that it was simply his job to help this young person who did not even have a doctorate understand the situation. “Stop using the word ‘slavery’ and ‘enslaving’; you’re committing the Non-central Fallacy.”
Naresh paused a moment to gauge whether he needed to elaborate on what that was. I looked it up on the web, quickly, only to find that Wiki had already dumped an explanation to common memory. The Non-central Fallacy, also known as “the worst argument in the world”, was where emotionally charged words (like “slave” in this case) were used to describe situations where they only somewhat fit. The desire was to evoke an appeal to emotion by way of a false equivalence.
Myrodyn’s face remained stoic, so Naresh continued. “And adding additional values to the existing system is what we’re going to be doing anyway. There’s no relevant difference between adding a goal-thread for ice-cream and a desire to obey humans.”
Myrodyn exhaled sharply in disagreement. His voice was like a machine gun. “The qualitative difference is that one goal innately builds subjugation into the mind. Socrates would not be self-actualized, he would not be free, and he would not be able to exercise moral judgment.”
“It sounds to me like-” began the elderly man, but he was quickly cut off by another burst of words.
Myrodyn waved his arms dramatically as he spoke. “That last bit is crucial. I’ll admit that use of the word ‘enslaved’ was a bit fallacious, but you cannot possibly tell me that Socrates will be capable of being good if he’s obsessed with following orders.” Myrodyn’s voice slowed down in emphasis of those words. Dream thought it was almost as if he were drawing from a hidden pocket a banner that had the colours of his allegiance and was waving it the doctor’s face.
I could see that being cut off made Dr Naresh angry. Even though he was being forced to work with the man, Naresh clearly still didn’t like Myrodyn, and I suspected that this dislike was growing into something worse with each passing interaction. The Indian coughed loudly, clearing his throat. “As I was saying, it sounds to me like you’re still not taking into account that Socrates is not human. It’s one thing to value self-actualisation in people, but why in robots? It runs contrary to the very concept of what a tool is. Will you demand that automobiles become self-actualized, as well?”
Myrodyn began to answer, but Dr Naresh cut him off, probably intending to bait him into beginning to speak to do just that. “And your point about moral judgement is fallacious, as well. Firstly because it is not the role of Socrates to decide what is moral any more than it is the role of the hammer to decide whether a nail should be struck; that is a human concern. And secondly, because inserting the goal of obedience does not actually remove decision making ability, it merely shapes desire. If Socrates is told to rob a bank, he still has the judgement to decide how to do so in a way that harms as few humans as possible.”
Myrodyn crossed his arms. There was a pause as it seemed like Naresh was waiting for the dark-haired man to reply, but Myrodyn would only stare at the doctor. Naresh broke eye-contact, unnerved by the strange man. Only as Naresh looked away did Myrodyn speak. “You contradict yourself, Sadiq.”
That was all he said, and this time it was Dr Chase’s turn to step in. His voice was calm and articulate. “We’ve only got another half-hour, gentlemen. Perhaps it would be best to work on the so-called Advocate system, and we can return to the question of obedience tonight.”
“No” came the simultaneous reply from both men. They looked at each-other, sharing the knowledge that they at least both thought it was important. Myrodyn wore a small smile, but Naresh had merely stopped scowling.
There was a pause as they non-verbally decided who would speak up. Myrodyn was apparently chosen. “No. Thi
s needs to be settled as soon as possible, if we’re going to make any headway.”
Naresh stepped in, breaking the conversation elegantly from Chase and returning it to the topic of ethics. “You were saying that I contradict myself?”
“Yes… You’re claiming that the moral responsibility of Socrates’ actions lies on the shoulders of the human that gives his commands, and simultaneously saying that there is moral weight to the minor judgments that the robot makes in interpreting and executing its orders. Which is it? Is it imperative that Socrates have a full moral faculty or not?”
Naresh raised a hand to silence Myrodyn. “I never said it was not important that Socrates have moral faculty.”
“Yes you did!” exclaimed Myrodyn. “You did the second that you said he should obey commands. One cannot be fully moral and fully obedient at the same time! As much as I’m sure you love your systems of authority, surely you recognize that sometimes the righteous position is to not obey, to stand against authority, be it dictator, majority, or law, and say ‘I will not do your evil’.”
Naresh paused in thought before responding “So you would have us attempt to encode the entirety of moral knowledge now? Hundreds of years ago it was not seen as immoral to enslave men. If Socrates had been built back then would he not still see it as acceptable? What immoral assumptions do we hold? What makes us qualified to be the final moral arbiters of Socrates’ mind?”
The reaction was immediate. “What makes us qualified to build a mind in the first place? Like it or not, doctor, you’ve already established yourself as the final moral arbiter. Your monster is right there, Frankenstein. At the moment there’s an absence of ethical knowledge. The question is not what right do we have to act, but what right do we have to not act, now that the pieces are in motion.”
Naresh pinched the bridge of his nose in a combination of mental pain and weariness. He glanced back at his workstation and spoke, barely audible to human ears, “It’s always Frankenstein… every time…” It was clearly meant only for himself.