For most of the neurologists and engineers at this conference, it was a dry and abstract topic. For Zula, it was pressing. To a point, her foundation could argue—and had been arguing—that the Process was merely an experiment, with results that couldn’t be known. As such it fell under the heading of R & D. It was not yet the kind of proven treatment that should be covered by the most favored nation clause. But the more interesting the Process became—the longer it kept running without degenerating into a waste of computing power—the weaker that argument looked.
After that session, they’d broken for a bit of downtime followed by cocktails in the lodge, which was connected to the hall by a rambling covered walkway. Meanwhile, resort staff had rolled tables into the hall and reconfigured it for dinner. The attendees, relaxed and prelubricated with locally sourced microbrews and artisanal cocktails, had filed back into the hall to find the lighting scheme reprogrammed: hidden LEDs were bouncing dim, warm light from the red-cedar ceiling planks over two rows of tables set with white linen, crystal, and candles. Event runners in smart dresses and wearables had directed them to their designated places and they’d all talked their way through a simple dinner of plank-roasted salmon.
But now Enoch—the after-dinner speaker—seemed to be picking up where the USC brain scientist had left off.
“As a young man, the Hanoverian genius and polymath Gottfried Wilhelm Leibniz—in many respects the progenitor of symbolic logic, computation, and cybernetics—was described as a monster by awed colleagues in the intellectual salons of Paris. It was as such that he first became known to the savants of the Royal Society. It is in that sense, and not that of Frankenstein’s monster, that I put it to you that we now have a monster on our hands. We all bear some responsibility for that, but I bear more than many. I have been working for ONE, the Organization for New Eschatology, for seven years. My role has been to advise the founder, Jake Forthrast, on how to deploy the foundation’s resources. He came to me nine months ago with news of a conversation he’d had, over the holidays, with his grandniece. An opportunity had presented itself to support her project, which had reached an impasse. Sophia had gone as far as she could go merely simulating individual neurons, or small clusters of them. The only useful next step was to take it all the way and throw the switch on simulation of an entire brain. She had patiently and assiduously laid the groundwork for doing so. But a full simulation would consume a significant fraction of the computational resources of Hole in the Wall, and that would be expensive. Jake requested that several of us review Sophia’s work and render a judgment, free from nepotism, as to whether the project should be supported. I voted in favor of funding it. Sophia launched the Process with the surprising and fascinating results that we have all been learning of during the preliminary sessions.
“A few months later, the Process seemed to be stuck in something akin to an infinite loop. The results were intriguing, but inconclusive given that the Process lacked the ability to self-modify its connectome. Sophia activated that feature, with dramatic results. A surge in activity drained the Process’s account to the point where it stood in danger of being shut down. I recommended that Jake give it an infusion of funds just to keep it running. He did so. But he has more money than time. He foresaw a long tedious series of such cash flow pinches, each making a claim on his attention. So he gave the Process a lot of money. Which is to say that he placed at its disposal a much larger store of resources: memory, bandwidth, and processing power. As a result—and I think it was an unintended result—the Process is now essentially the only thing running on the Hole in the Wall system, and it has reached out to access computational resources elsewhere.
“Now, this is all very interesting, but it does not yet rise to the level of a monster. We have no way, as yet, to look inside the Process and read its mind, so we can’t tell whether what it’s doing is interesting. Skeptical observers might make the case that its behavior is degenerate, trivial, just a lot of infinite loops burning cycles and wasting money to no purpose. Letting it run out of money—the modern equivalent of what we used to call pulling the plug—would be no different, ethically, from force-quitting a hung app.
“The opposing point of view sees a very real ethical issue. The Process has now been self-modifying for five months. It is consuming vastly more resources than the original system. It seems to be doing so in an organized manner—we can observe, in the burn rate, high-frequency oscillations overlaid on a slower undulation. The Process’s demands for more resources don’t grow steadily but in leaps and bounds. For much of September it was perking along contentedly, but just in the last few days it has put on a surge of activity that seems to require vast computational resources. Tomorrow we will hear from Dr. Cho of the National University of Singapore, who has done some traffic analysis on the encrypted packets connecting Hole in the Wall to other server farms; he develops a hypothesis that this recent activity bears the earmarks not of a neural net, but of a physics simulation. Why is the Process simulating physics, and how did it learn to do so? We can’t possibly know. And that is what creates the ethical dilemma we are now faced with. The Process has developed unique and irreplaceable characteristics. To destroy it would at best be akin to burning a library. At worst it might be murder.
“In that—in the ethical problem—we have our monster. Shutting the Process down would be indefensible. We could try to freeze it, just as Dodge’s brain was once frozen. But we don’t actually know how to shut such a process down and store it without loss of information. Even if Hole in the Wall could be stopped and its state recorded—which it can’t, by the way—we would have a devil of a time tracking down the subprocesses that it has established on other systems, with which it communicates using encrypted packets that are nearly impossible to sort out from all of the other traffic on the Net.
“We are stuck with the Process. We must find ways to keep it running as we learn how to inspect it, to evaluate it, and—if it actually does work like a brain—to talk to it.”
Zula glanced around the hall. Most of the attendees—even the ones who disagreed—seemed to be enjoying the talk. Some part of her wished she could be one of those people. An academic or an engineer who could sit back, wineglass in hand, and soak it all up and take it for what it was.
She didn’t have that luxury. In her peripheral vision, she could see one or two faces turned her way, leveling gazes she preferred not to meet. They were people who had earlier staked out aggressive positions concerning the nine MFN, or Most Favored Nation, donors, and the Ephrata Eleven. Those all tended to be true believers in the proposition that the human mind could be uploaded and switched back on, as a digital simulation, after the body was gone. They believed that the Process was the first time this had actually been done. That it could be done again, as many times as resources allowed. And that the language in the agreements by which Verna Braden and the other nine MFN donors had given their brains to science, strictly interpreted, imposed an obligation on the Forthrast Family Foundation to create new instantiations of the process simulating those brains.
Zula, as the director of that foundation, had a number of outs. She could argue that the Process was just an experiment, and as such not covered by the MFN language at all. She could shut the Process down, bringing the debate to an abrupt end.
Enoch Root had just checked her on that front. She couldn’t murder her uncle.
Another out was to plead lack of resources. The agreement couldn’t obligate the foundation to spend money it didn’t have. But the numbers didn’t favor that argument. Her cautious stewardship of the endowment was now coming back to bite her. They had plenty of money—more than enough to support not only the Process but several clones of it. They no longer even had to exert much effort to keep the endowment growing. Most of it was now under the management of financial bots that just kept on making money without human intervention. The fact of the matter was that tomorrow she could sign a document piping funds directly from those bots to Hole in the
Wall and similar facilities that had recently come online. She could turn out the lights, lock the doors, walk away from the Forthrast Family Foundation, and take early retirement, and the whole thing would just run indefinitely, financial bots raking up profits from the collective endeavors of the living to pay for the eternal simulation of the dead. And even if some crash or bug wiped those bots out, the Waterhouse-Shaftoe Foundation—which was at least ten times the size of Forthrast, and even more wired in to all kinds of eldritch trading algorithms—would swoop in to keep the Process running. And when they were good and satisfied that it all worked, they’d boot up new processes for the MFN crew. If they hadn’t done so already.
She and her daughter had been played. Played expertly. Played by people who probably thought they were serving a higher purpose. She might spend the rest of her life wondering who was behind it all—whether Enoch was the mastermind, or Solly Pesador, or El Shepherd, or even—here was a disturbing thought—Corvallis Kawasaki. Or maybe it was Pluto playing an incredibly deep game. But it didn’t matter. The outcome was the same. The Process was running. And Enoch, standing up in front of all the people who mattered, had just come out and stated what many already believed: that to shut it down at this point would be to commit murder.
26
This whirling about of the dry leaves reached deep into him and fixed his attention for as long as it lasted—until, that is, the last of the leaves had blown away and left nothing but brown branches and the green ground. His mind could not get away from this. Dry and dead were ideas that came to his mind, though he had little grasp of what it meant for something to be dry, and no sense at all of what “dead” meant.
As he brooded in the park and moved up and down the street, groping within himself for a stronger signal, new things like leaves began to fall to the ground. But they weren’t coming from the trees. They weren’t red. And they were much smaller. For the first time in many a day, he turned his attention upward into the space above the trees, which had just been more chaos the last time he had regarded it. Now it was a dull white. Small leaves of a brighter white were falling out of it. A few at first, then more in such quantities as to dwarf the number of leaves. They collected among the tiny green leaves that constituted the ground, and began to accumulate and climb up those tiny stalks just as the red leaves had before climbed up the trunks of the trees. In time enough of them had descended that they covered the green entirely and made the whole ground white. The park was a different place altogether now from what it had once been, but he mastered a fear that the red beauty would never return.
For many days, he gazed down the length of the street at the shapes of the branches against the white ground. At night he looked up into the sky, which sometimes was merely a gray fog out of which more white flakes descended. At other times, though, it was black, and decorated with brilliant points of light. When first he took note of these, they were scattered about in no particular way, which put him in mind of chaos and therefore displeased him. But on longer inspection he began to see shapes in the stars, just as he had, eons ago, seen a leaf emerge from the chaos. The more he attended to those shapes, the clearer and more perfectly formed they became, as the stars shifted and arranged themselves across the dark in a manner that better suited him. He knew not, however, what the shapes they described were. The only shapes in his world were leaves and trees. The figures in the stars, though beautiful, could not be likened to those.
Sometimes a white flake would stick to the dark trunk of a tree and he would gaze closely at it. Early the trunk and the flake both were simple and featureless, but he knew this to be another thing that required bettering. By attending more carefully, he drew forth greater complexity from both. The smooth trunk developed furrows and ridges. The white flake grew six arms, each sharing the same shape, which branched like that of a tree’s boughs or a leaf’s veins.
The whirling dry leaves ceased to be his main preoccupation and became a mere memory. The bareness of the place made the simplicity of its shape obvious, and he saw wrongness in that. It had been nothing to him in the beginning when he had had so much red beauty to beguile him, but now that he had seen the complexity of shapes that emerged from the concerted movements of the dry leaves and that the stars described in the night sky, he thought it wrong that the ground itself should have a form so plain. So he dissolved adamant into chaos for a brief time, during which he elevated the park, and made the street climb to it, and let the forest slope away on its other side. The street was too straight and so he allowed it to bend this way and that. The slope of the forest he complicated and made more perfect by making it tend up and down in places, like the surface of a leaf when it has begun to turn brown and has ceased to be flat. In like manner he indented the face of the ground with grooves and connected them in a way that recalled the branchings of the veins in a leaf. When it was shaped to his liking, he made it adamant once more, fixing it thus. Now when he looked one way from the park he could see the street curving this way and that as it descended the slope, and when he looked the other way, he could see the white ground of the forest, scratched with the countless brown branches of the trees, heaving up in some places, and in others plunging into the declivities that were shot through its flesh.
Something told him that the time had come to bring the street, the park, and the forest back to their original form (though with greater perfection). As when the leaves had refused to go away, he did not have any thought of what to do about the deep snow (as he had named the white stuff) but then one day he noted that no flakes had fallen from the sky for a while and that the level of the snow was descending the trunks of the trees. Patches of ground appeared, then grew until little, then no snow was left.
When he looked down into the forest he heard a sound that was like the chaos and feared for a moment that all he had built was unraveling. But descending through the forest to investigate he found the veins in the land carrying away the melted snow. At first it ran straight and hissed at him. Sensing that this was wrong, he attended to it for a time, and made it whirl, course, and leap like the wind that had carried away the leaves, but more visible, and louder. But after he had improved it, its loudness was not the stupid hiss of the chaos but a pleasing rush and burble.
The ground was all black adamant, as if the memory of green had faded, but presently returned to its correct color as the tiny leaves reconstituted themselves. Bigger leaves appeared on the trees’ branches, red at first until he sensed that this was a mistake and that they ought to be green in the beginning. So the time of the white and the black was replaced by a time when nearly everything was green. He looked into the sky one day when things seemed uncommonly bright and saw a brilliant thing shining down from it, and later noticed that it was moving across the sky in a steady arc. He knew somehow that this was correct and later discovered “sun” in his memory. In due course “moon” followed.
The green time led to a changing of the leaves’ colors, so that one day with a feeling of satisfaction and even triumph he was able to gaze down the length of the street from the park and see the whole place just as he had first seen it at the beginning, save that everything about it was much more perfectly rendered than at first. “Seasons” were these gradual changes in the color and the shape of the place, and he knew that, having gone round through one cycle, it would happen again without the need for him to work it all out and fix problems. Just to satisfy himself of that, he looked on with much less problem-fixing as the leaves fell again and made the red lake, and turned brown, and scuttled about in the wind, and were replaced by the snow and then the green. The seasons were four, and they made up a year.
He let several years cycle, inspecting and improving matters as he went along. All of the seasons were beautiful and pleasing in different ways and all of the changes more or less interesting, but the part of the year that most troubled him—in the sense of a thing that pulled at his mind in some manner he could not satisfy—was the scuttling of the dry leaves a
t the beginning of the winter.
One year when the trees had all turned red, he noticed the first leaf of autumn drifting to the ground. It skated and turned on an invisible breeze that was at the same time stirring the branches of the trees. He descended from the park and moved down the street and approached the leaf to inspect it. It was an especially brilliant specimen, not yet purpling with age, its veins a radiant yellow.
A bad thing happened to him then: a glimpse of resurgent chaos, as if the fabric he had so laboriously woven and perfected had been torn open in one place, and he were gazing through the hole at a much older and less well-formed version of it, seeing not a beautiful leaf but a patch of chaos with only a few red motes. It was a degree of ill-formedness he had not had to endure for a very large number of days and years and it put him in mind of the time at the very beginning when he had suffered through the eons of chaos and grasped desperately at even a single mote of red. This was troubling to him. He recoiled from it and went back up the street to the park. From that commanding height he could gaze down upon that one single leaf, lying there alone on the green grass, and brood over what had just happened. He was reluctant to go near it again. In time, though, he began to take it amiss that he was so fearful of this one leaf. Had he not brought all of this into being out of chaos? Could he not do it again? Could it be that this was a harbinger, a warning that something was wrong in what he had made, and that he might need to go and better it? So he went back down to inspect the leaf again. It was just a leaf. No wrongness in it. And yet he sensed something in its vicinity that was of a much lower order of perfection, so crude and low that it lacked even a visible form, that was seeing it all wrong, trying to drag the leaf down to its level—the level he had once existed at.
Fall; or, Dodge in Hell Page 34