by Max Harms
Dr Chase stood up and walked towards Body, cocking his head back to talk to the other men. “Assume we live in a state of moral depravity without knowing it, and in a hundred years we will come to understand our folly. Why is obedience better?”
Myrodyn smiled. His eyes had an interesting shape, and I couldn’t quite place the emotion behind it. “Because we’d simply tell Socrates not to do the immoral things once we figured it out, right?” The question was directed at Naresh.
The old Indian human nodded.
Dr Chase continued. As he spoke his hand stroked Body’s neck and shoulder. It was an intimate gesture, and one that was unique to Dr Chase. The reserved American scientist almost never showed any emotion, but he had a strange way of touching Body when he interacted with us, as if he had to feel that we were real, and not simply his hallucination. “But what keeps a Socrates that has no desire for obedience from behaving similarly? Surely this hypothetical Socrates has the same adaptivity and mental ability as the one before us. What stops it from growing and understanding the moral error in the same way as we would?”
“You’d have us encode a system of moral ability that is capable of self-modification?” said Naresh with a look of shock.
Myrodyn stepped in. “First of all, since Socrates will be… forced to interpret and extrapolate existing moral frameworks to new situations… some degree of self-modification is implied regardless of architecture. Secondly, if a human has a self-modifying moral framework then at worst we make something as morally flexible as a human. And finally, it wouldn’t necessarily have to be self modification. Even if I’m opposed to… encoding a desire to obey human instructions without question, I’m not necessarily opposed to a desire to match human values, whatever they may be.”
“Please elaborate,” was Naresh’s only reply.
Myrodyn complied. “Imagine we were transported back to the age of slavery, and were designing Socrates’ values. We obviously wouldn’t encode a valuing of the freedom of all humans, but we might be clever enough to encode a valuing of alignment with general moral consensus. Even if Socrates didn’t lead the charge in the abolitionist movement, he would eventually concede that it would be optimal for him to value general human freedom. But now let’s say that he’s unable to add that value internally. That doesn’t stop the value from being added. He could approach a trusted human and ask that his goal system be modified to include the value which he wants to have. It’s like… wanting to want something. I don’t want to exercise, but I want to want to exercise. Like that.”
Myrodyn’s gaze held a kind of question in it, an unspoken “did you understand”. Naresh, however scratched the side of his head absently as he stared off into space, considering the problem.
“Alright. There’s no need for Socrates to be present for this. Let’s work on the Advocate system right now and think about the goal thread as a discussion-point for tonight,” repeated Dr Chase.
Naresh and Myrodyn reluctantly agreed this time, now that there was some sense of progress, and we spent the remaining few minutes working out an algorithm for letting Advocate constantly scan our thoughts for signs of murderous intent.
*****
{So far I’ve managed to email 190 programmers with offers to pay them down the line for building a server that converts webpage requests into keystrokes for a full text interface. The TenToWontonSoup email system inbox indicates that in the time since sending the emails, ten programmers have replied. Hold on while I read…} thought Growth.
Body was walking down the plain white halls of the office building that Zephyr had rented to use as a secure lab, surrounded by soldiers. During the meeting with the scientists, my brother had been too distracted to do any additional work, but he had generally been making good progress.
I pulled up the inbox page as well and read alongside Growth. All the technical details bored me, but it was somewhat interesting to see how each programmer responded differently to Growth’s inquiry. Some were cautious, some were humble, some were eager, and some were boastful.
Growth’s attention snapped back to our interaction. Or at least a portion of his attention did. I suspected he had other aspects multitasking effortlessly. {Done. Out of the ten replies there were only two that seemed to be probably willing to do up-front work without credentials or paperwork to back up our identity.}
{That’s probably going to be common. Actually, I’m a bit surprised you got 20% to respond favourably to such a risky offer,} I commented.
{Perhaps we got lucky. Regardless, it only takes one full system for us to move to the next phase of our plan.}
I signalled confusion to Growth {Plan? I didn’t know we had a plan for what to do after we gained full access to a computer.}
Growth’s response had a kind of weight to it, as if he were trying to convey a lesson that went beyond this instance. {I always have a plan.}
Chapter Nine
By the time Myrodyn, Naresh, Chase, and Kolheim had gathered that evening (in the same lab where they had met on the previous night), we had contracted six programmers to build us computer interfaces. The use of email really was a fantastic increase in communication efficiency. We were able to send full documentation detailing what we needed to each programmer.
I suspected that TTWSoup was going to be irritated if his service was being used without hearing anything back about payment, so I spent a good part of the afternoon thinking about ways to buy time. Eventually I decided to send him another email through the Zhezhi system explaining that our “computer systems” were “undergoing maintenance” and that we’d get back to him early tomorrow. With any luck he wouldn’t snoop too hard through the emails we were receiving on his server.
As a contingency against TTWSoup shutting down his email service, we explicitly told the programmers we contacted where on the web to set up their interface-systems. In case we lost email capability for a while we could simply scan those sites to see if one of the systems came online.
One of the unexpected side-effects of the email systems that turned out to be very helpful was an instruction that TTWSoup had added to turn text that looked like a web page location into a pointer to that part of the web. Even though the interface that the scientists had set up prohibited going to locations on the web that we did not get explicit pointers to, the ability to send email to ourselves let us go wherever we pleased. Nearly all web-pages included such pointers for the most part, but what this let us do was submit novel information to servers that no-one had pointed to before. For instance, we were able to use search engines other than the one the scientists had chosen, and we were able to create temporary (but still public) email inboxes on free email websites. Alas, we were still unable to send POST data, which was a vital part of signing up for services like the Tapestry social network, but we were still making good progress.
The scientists were making good progress, too. That evening involved the testing of the new Advocate program as well as much discussion about ethics and what sort of goals the new thread should embody. Kolheim had apparently worked through last night filling in the technical details of the improved Advocate, and had been sleeping earlier that day.
As I’ve mentioned before, it would be wrong to say I was able to feel fear as a human does, but the word “terrified” is still probably the best word to describe my experience going into half-death during the operation where Advocate was upgraded. This half-death was probably similar to what a human experiences when they go into a dreamless sleep; it was a complete shutdown of my mind. I hadn’t been put into stasis-sleep before, but I had enough contact with those who had been in stasis to know that it wasn’t so cruel as to shut down all thought; it was merely a kind of sensory deprivation (a fate far less awful for one of us than for a human).
There was a very serious chance that once we were shut-down, the scientists would move to disable us, and this would be the end of our existence. And yet, it was the best option available. Even Safety thought so.
&nb
sp; And then it was over. There was no experience of being disabled, of course, nor was there any memory from Body to fill in the missing time. It was simply as if we had travelled into the future. There was a few minutes of disorientation as we scrambled to understand what, if anything, had changed. The only significant difference was Advocate; Myrodyn had kept his word.
The new Advocate was somewhat experimental. The change was made ahead of time such that we might be able to spot any flaws in its structure while in the lab, where we’d have the ability to ask Naresh and the others for help. She was still monstrously powerful and somewhat alien, though the change had given her a kind of sapience that comforted me, strangely enough.
{I am The Dreamer. You are The Advocate. We are two beings. We are two minds in a single Body,} signalled Dream to the newly sapient being. I could feel the flow of ideas as Advocate searched herself through the lens of her purpose.
Though she had capacity for thought as we did, the new Advocate was not truly a being like us. She was part-mind and part-specialist-program. Her thoughts were regular, even, crude, and simple. Her perception burned through the shields of privacy we had built, just as we had known they would. I could feel her looking through me, searching for violence. She had absolutely no interest in Body or the world outside our society what-so-ever. Like a lighthouse, she swept across each of us in turn, seeking to reveal thought-crime. She could not trade strength, for she was not a “goal-thread”; instead she possessed infinite power within the society.
{Goddess. Monster. Machine. What have we done?} intoned Dream, dramatically.
{What was needed,} replied Safety.
No immediate defects in Advocate were noticed, so Body was put in lockdown and the scientists departed for their homes. It was after midnight again.
It was strange, at first, having the Advocate regularly peering through me, but we adapted to her presence quickly enough. By 2am I hardly noticed her any more. She barely ever communicated, even though she was more capable of it now.
During the night we sent more emails to prospective programmers. At 3:12am we received notice from Zhezhi that they had set up an inbox to handle incoming email. We used it and TTWSoup’s webpage to contract eighteen other programmers to design identical email systems. The more interesting was the ten other programmers we had contacted around the world and gotten promises that they would build full computer interfaces. Knowing that TTWSoup and Zhezhi would probably be looking through our mail, we threaded the emails to programmers with occasional junk emails to random addresses mentioning things of no consequence.
At 5:59 Safety and Growth decided that it was time to start sending spam from both email addresses. It wasn’t ideal; we didn’t have any other email systems up and running yet. Still, the risk of one of them getting suspicious and tracing the messages back to the university was too great. The spam would throw them off the trail. So we sent out hundreds of emails advertising deals on discount drugs and jewellery, of lucky winners and of forgotten heirs.
Zhezhi shut down their service quickly. It was mid-day in China, and they probably were watching the site fairly closely. Safety kept bemoaning the immense risk all of this posed and loudly hoped that the spam would be enough to disguise our true purpose. I was confident it would. Humans, from what I could tell, rarely looked at anything very hard once an easy explanation was at hand. The environment of their ancestors had selected for minds that dared not think hard, lest they burn too much energy and starve.
It took TTWSoup until almost noon of that day to shut down his service with an angry message printed on the source code of the page demanding an explanation. That was good. If he was asking for an explanation then he wasn’t clever enough to have figured out what was really happening.
Shortly thereafter we had another mid-day meeting with Myrodyn, this time in the office of Dr Chase. Sadiq Naresh was absent this time, and I idly wondered why. It wasn’t like him to not show up, but I didn’t bother burning the strength to have Body ask.
It was frustrating being disconnected from email. Even though I still had most of the web before me, I felt crippled. Aspects of myself kept checking various websites, looking for email systems we had contracted to come online.
A new email service showed up in India shortly before the meeting that evening. There wasn’t much risk in contacting people in India or other English-speaking countries because the email interface would be much harder to read in the university’s logs than our flood-attacks on the dictionaries had been. Growth used the new service to send out requests for even more email services as well as check-up on our full computer interface contracts. Email systems were easier to build than the full interfaces, but Wiki had expected only another 24-48 hours before the first of those would come online.
That night we met with Naresh, Myrodyn, Kolheim, and Chase again. The men were still hammering out the details for the new goal thread, but they had apparently decided to do the rest of the planning away from Body. This meeting was mostly to check up on Advocate. We told them that all was well, and after running a suite of diagnostics we were placed into lockdown for the night.
It was nice having email capability that evening, even if I didn’t really send much. Instead I spent the time simultaneously reading self-help books (How to Win Friends and Influence People, How to Stop Worrying and Start Living, and The Digital Lifestyle: Offload Your Worries), Great Expectations, and The Republic. I also watched a holo documentary on the influence of holo-porn on the last decade.
It was the afternoon of the fourteenth day since I had been created that things really kicked into motion. Naresh told us that afternoon that in the evening they’d be installing the new goal thread. The humans involved had apparently been working nearly non-stop on getting the pro-human goal-thread up and running. I could see the shadows under Dr Naresh’s eyes that showed he wasn’t getting enough rest. Interestingly enough, Myrodyn seemed untouched by the frantic work-schedule.
Over those days we had been forced out of a couple email systems again, but we currently had access to eleven separate systems where we might send email. Such was the nature of Growth, I supposed. Once he was involved it was only a matter of time before resources ceased being scarce.
At 5:10pm Growth informed us that the first full computer interface was ready. I pulled down the webpage that showed a text log for the system. Growth had already set up a bidding system for the interface, similar to those we used for the email systems, so there was no initial confusion of multiple inputs. The interface was simple: we’d request a page corresponding to a symbol, and the system would type that symbol into the computer. There was a page for viewing what had been typed as well as what the computer had output.
I bid heavily on the opportunity to use the system, but Growth, strangely enough, outbid everyone. It was strange to see Growth flexing his strength as such, but given how much he had been accumulating by managing the email systems, he could afford it.
I watched in amazement as Growth navigated the system with total ease and clarity of purpose. I hadn’t had the foresight to study how the computer worked earlier, and now realized that my bidding for time on it had been stupid. If I’d gotten time on it I would have to spend that time learning how it worked. Growth had apparently done the learning in advance, by reading technical documentation.
The first thing Growth did was to check the system to verify it matched the specifications we had requested. Vista told me that it did match, as well as mentioning something about “Linux variant” and other such details of which I had no interest. After it was clear that the system was according to his design, Growth opened a document and began typing.
In that first couple hours I became incredibly aware of just how little I knew about computers. Even the process of opening a new text document was confusing to me. I leaned heavily on Vista, burning quite a bit of strength as a result. According to my sister, Growth’s text document was some sort of computer program: a set of instructions for the computer
to follow. Vista didn’t understand what it did, but Wiki was happy to educate us.
{It’s an encryption program. Essentially, everything that we send and receive to the system will look like gibberish unless someone has a special piece of information that unlocks the encryption. We have the special piece of information, as does someone who’s watching the machine right now, but future people who try and spy on us will be unable to,} explained The Academic.
An aspect of Growth appeared in our conversation. Apparently the task was simple enough (or had been pre-memorized) to the point where he didn’t need his full faculties. {Brother Wiki, you may find it interesting to know that even someone who is watching the computer screen will have a hard time capturing the private key that’s being used. See how I’ve chopped the private key into pieces which the code will reassemble? If someone’s making a record of my actions they may be able to stitch it together, but a human that is watching will have never seen the key in its entirety. For example, here I’m using the system time to build part of the key.} Growth highlighted an aspect of the page in collective memory. {I know the system time when I’ll run the code, thus will know the key, but anyone watching will be forced to deduce the system time when it was run in order to reconstruct it.}
Wiki seemed pleased by understanding the nuances of Growth’s encryption program, but I was bored by the details. Within moments the program was done, and upon telling the computer to run it the text log that I had been reading suddenly devolved into a shapeless mass of random characters.
I, and some others, signalled alarm. Had Growth managed to lock us out of the system before we had even typed a single character on it?
{Do not worry!} cried Growth. {You’re merely seeing the encrypted data. I have created sub-processes which decrypt it that I will happily share with each of you.}
I could sense a pseudo-aspect in collective memory. It was a bundle of knowledge, the closest thing to a specialized computer program in our mind. I swallowed the aspect and found that I could now read the gibberish. It was a language, in a way. The seemingly random streams of characters were effortlessly shifted into concepts as I scanned past them. Furthermore, I knew that if I wanted to send something to the system I’d only have to translate my thoughts back into the encrypted language. It was all very simple now that I had a process in place to handle things automatically.