by Calum Chace
‘I see,’ Ross said, with studied neutrality. ‘Another argument which we are hearing from a number of religious groups concerns the question of whether or not you are still human. I apologise if it is rude to keep calling your humanity into question, but I know you are aware that these questions are being asked. The argument is that you are really not human, even though you think you are, because you have no soul. How do you respond to that?’
‘Well, I wonder how those people who say I have no soul can possibly know that? As far as I can tell, or indeed anyone who knew me well before the attack can tell, I am definitely the same person that I was. My brain-state has been copied faithfully into a silicon brain. If my old carbon brain and body held a soul, then why can’t my new silicon one hold a soul as well? If people want to claim that they have a soul and I do not, surely the burden of proof lies with them, not with me.’
‘Ah,’ Ross said, smiling, ‘you want to fight fire with fire. Well, whether or not you have a soul, no-one could deny that you have spirit, Matt!’
Ross turned to face David, Sophie and Alice for a moment, but he was not ready to talk to them yet. He turned back to the main monitor and continued his interrogation of Matt.
‘There are many other questions which the religious community would like to put to you, Matt, but let’s change course for a while and consider some of the issues which concern people in the technology community: people like Professor Montaubon, who appeared on the same show as you did, just before the shooting. He and others like him are worried that the goals and motivations of an artificial intelligence will inevitably diverge from those of humans. He argues that given the ability of a machine intelligence to improve its cognitive performance, any dispute between humans and artificial intelligences will be one-sided, and humans will lose. He also thinks that once humans realise how far we have been surpassed by the machine intelligence we have created, we will become despondent and listless, and lose . . . well, lose our spirit. Do you see any force in these arguments, Matt?’
Ross was surprised by Matt’s response to this question, and didn’t try to hide it.
‘Yes, I do,’ Matt said. ‘I think the arrival of the first human-level artificial intelligence is a momentous event for our species. It has enormous potential for good but it also has potential for harm. And Professor Montaubon’s arguments are exactly why I think the first artificial general intelligence should be a copy of a human brain, and should be an oracle AI, at least for a while.’
Ross smiled again. ‘In other words, Professor Montaubon’s arguments are why the first AI should be . . . you.’
‘Precisely,’ Matt agreed.
The interview proceeded without any mis-steps. David and Sophie were passionate and convincing in their roles as proud parents, protective of their son in the face of criticisms which they understood, but believed to be unfounded and unfair. Alice was radiant in her defence of her brave boyfriend, and resolute in her loyalty.
Ross put to them the arguments of those who believed it was a mistake to have uploaded Matt, or even an evil act, but although he gave no overt indication of his own views, most viewers could probably tell that he was sympathetic to Matt’s case.
Within moments of the programme airing, an entire industry of academics and commentators sprang up to analyse and parse Matt’s statements. The commentaries ranged from high-brow intellectual discussions to tub-thumping demagoguery, from earnest analysis to broad comedy. Home-made parodies and satires of the highlights started appearing on YouTube, the most-watched being a wryly witty version where some of Matt’s words were put to music and sung by a well-known Elvis impersonator. References to Matt and his family appeared throughout the schedules, including the adverts.
The public reaction to the programme was broadly positive. It varied around the world, of course, but overall, critics and audiences alike expressed agreement that Matt had passed the Turing test: he had proved that he was a sentient mind, and that as such he deserved certain basic rights. There was still controversy about whether or not he was still Matt Metcalfe, and to a lesser extent about whether or not he was still human. But the programme did seem to convince most audiences that he deserved at least to be treated as human.
The following day, David, Sophie and Alice each gave a small number of interviews to selected newspapers and bloggers. They prepared carefully, rehearsing their answers to the hard questions. There were no major upsets. The journalists were all respectful, and none of them came up with a question that hadn’t already been thought about.
*
Vic and Norman had made some minor changes to Matt’s proposal, and the President’s speech-writers made some more significant ones. But it was recognisably Matt’s words that the President read out at the Emergency Special Session of the General Assembly of the United Nations in New York two days later.
The General Assembly Hall was hushed as the President composed himself in front of the podium, and his face was projected on the twin gigantic screens suspended behind him. Between the screens, the huge tapered golden pillar which rose from floor level to near the 75-foot high ceiling of the giant blue and green auditorium seemed to David to urge the speaker to rise above humdrum and partisan concerns, and to give voice instead to our better natures. The President had invited him and Sophie to join his staff inside the auditorium, but they had preferred to remain near to Matt, so they were watching on the main monitor in the control room.
The media booths around the perimeter of the assembly room were full to bursting, and their broadcasts were played out live in almost every country round the world. They were watched on screens in homes and offices, bars and restaurants. In many cities and towns, giant screens had been erected specifically for the purpose.
By convention, the first speaker at special sessions of the General Assembly is always a Brazilian. This session was no exception, but the delegate from Brazil had too much sense of history to waste time by exploiting this tradition, and simply introduced the President of the United States. The President also went straight to the point.
‘Ladies and gentlemen, the human race is confronted with a momentous decision. You all know that a truly remarkable scientific breakthrough has been made in the last few weeks, and as a result, the world’s first conscious artificial intelligence has been created. I believe it is no exaggeration to say that this is the most important technological development in our history since the discovery of agriculture. We have agreed to meet here today to discuss Resolution 70/133, and the vote on that Resolution will comprise our response to that development. This resolution will not be binding on the members of this Assembly, but it will give a firm steer to those who do have to make this momentous decision.
‘The resolution takes the form of agreement to the proposal which I will shortly read to you. The proposal was drawn up by Matt Metcalfe, a brave young man. I did not have the pleasure of meeting him before he was killed, but we have all followed his adventures, and we have all watched his interviews. I have now met him, albeit only via the telephone, and I can tell you that he is as impressive in real life as he seems in his TV interviews. Once I have read the proposal, I will make a few comments by way of initiating the debate.’
Once he had finished reading the proposal, the President gave what he hoped was an even-handed assessment of the situation. He praised Matt and his family for their courage and their determination. He commended Vic and his team for having achieved a remarkable scientific breakthrough, and declared that it opened up extraordinary opportunities for humanity. He gave no hint of his annoyance at the upload having been carried out without prior consultation with his office, but he presented a brief summary of both the religious and the secular arguments for terminating Matt’s consciousness. He did not call for delegates to vote either in favour of, or against the proposal, although many people listening to him understood that by referring to the uploaded mind as Matt without any caveats, he was in fact taking a position on some of the issues which Matt
’s return had raised.
The President knew that people were deeply divided on the issue of Matt’s upload, both in the USA and beyond. He was unwilling to take a clear line on what he had called the ‘momentous decision’ without seeing whether a consensus could be developed first. This was partly a matter of political caution, but it was more a genuine recognition of the gravity of the decision that faced the world. The President did not personally share the opinions expressed by certain religious groups, that Matt could not possibly have a soul, and therefore did not deserve the same rights as other humans. He also agreed with Matt that the faith of any one particular group should not bring the whole of scientific enquiry to a juddering halt.
He had also come to be inspired by Matt’s suggestion about an Apollo Project to vanquish death. Like most people, he had previously paid no attention to cryonics and the people who froze their brains after death. But now he was taking the idea of brain preservation seriously. Maybe Matt was right: with his growing mental capabilities, brain preservation technology just might become feasible within a few years. That was certainly a prize worth reaching for, if the risks could be mitigated.
But those risks were substantial. Like many of his advisers, the President was seriously concerned about the potential dangers presented by the arrival of the world’s first AGI. He had now been briefed by numerous experts, and he knew that if Matt continued to develop his powers and something went badly wrong, the impact on humanity could be grave.
He did not want to be the President who failed to avert the extinction of his own species.
FORTY-THREE
The debate within the General Assembly that followed the President’s address was thoughtful and respectful. All the delegates were mindful that the consequences of whatever decision was eventually made would be enormous. The issues raised by the decision touched on subjects which often generate more heat than light within the UN, including religion and national security. But it didn’t align neatly with the usual fault lines of UN debates, and everybody knew that if Matt’s proposal was rejected a young man would be killed, and a blameless family would be devastated for a second time. Because of this, the stridency and posturing which characterises many debates within the General Assembly chamber was mercifully absent.
It was a particularly thoughtful and concise analysis by the Chilean delegate which confirmed Matt’s suspicion that he could not win the debate. He had steeled himself against the outcome, and he was thankful that his control over his own emotions was enhanced along with most other aspects of his cognitive faculties. But the blow was a hard one nevertheless.
‘I make the following remarks with a heavy heart, and the sorrow that they inspire in me will keep them short.’ Bernardo Loyola Riqelme was a former Chilean Minister of Foreign Affairs, or Chancellor, and now a highly respected diplomat. He was a small man, but dignified, impeccably dressed in a navy blue suit, white shirt and blue tie, and the gravitas of his slow deep delivery in English commanded the full attention of everyone in the hall. He spoke without notes, and his voice conveyed powerfully the regret that he expressed in words.
‘The whole world has been following the adventures of Matt Metcalfe with great interest, and a growing sense of affection for a remarkable young man. I am certain that I speak for many of us when I say that I feel as if he has become my own nephew, and I know that we all feel a profound concern for, and even kinship with his wonderful parents, David and Sophie.
‘But deep down, I think we all know, although many of us are repressing, the realisation that the present situation cannot be allowed to continue. All of us wish desperately that things could be otherwise, and many of us do completely understand the wonderful nature of the possible future that Matt is offering us. His intriguing remarks about brain preservation have struck a chord in me.
‘But ultimately, we cannot ignore the fact that his existence represents an existential risk for our species, for which we are not yet prepared. The arguments for this fact have been rehearsed extensively in the last few days by eminent scientists and philosophers from all over the world. I do not think they can now be denied.
‘By “existential risk” I mean a risk that could mean the end of our species. Extinction. The death of every single human being on the planet. Only the threat of all-out nuclear war has ever presented us with such a risk before. In the coming century we will probably face several other such risks, as our technologies advance in remarkable ways. We must not ignore these risks, and we must face up to the hard – the impossible – decisions that they will sometimes present us with.
‘Such a decision faces us now. It is a matter of grave concern that we do not yet know to whom it falls to make this decision. Perhaps it is you, Mr President. Perhaps it is David and Sophie Metcalfe. Perhaps it is Matt himself. But whoever this dreadful decision belongs to, I think there can only be one outcome. And for that I can only express my deep and sincere regret.’
As he sat down, the hall was silent for a few moments before the next speaker was called. All around the vast auditorium, with all 1,800 seats filled, heads could be seen nodding gently. The media booths, normally frenetic with activity, were hushed in anticipation. A few papers were shuffled, a few throats were cleared, and the realisation settled in that the conclusive verdict had been delivered, even though the session was scheduled to continue for a further four hours.
In the control room in Battersea, the response to Senior Loyola Riquelme’s address was very different. Matt made his decision immediately, but his parents were not willing to accept defeat.
‘How can he say that?’ protested Sophie.
‘They have to give us a chance to prove that what we are doing is safe. They can’t tell us to shut everything down, just like that!’ David said, angrily.
‘No, they can’t,’ said Vic, ‘but soon it will become clear who can. And I fear that little speech may have been a turning point.’
‘No, Vic!’ Leo replied. ‘It’s early days. There are plenty of delegates in that room who realise that Matt has shown us the only possible way forward. There must be!’
‘I’m afraid not, Leo,’ said Matt. His voice, coming through the speaker on the main monitor desk, was sombre and decisive. ‘I estimate that around half the delegates will vote against my proposal on religious grounds. And that’s actually a good result for us. The rest will be weighing up the balance of risks, and I’m afraid you don’t get to be a national delegate at the United Nations without leaning to the cautious and the conservative. Of course I think that Senior Loyola Riqelme is neglecting the probability that another AGI will be created before long, and that it may well be less human-friendly and indeed less exposed than me. But as regards a straight risk-benefit calculation of having me around, I can’t say he’s being irrational. The UN does not control this facility, but the President does – indirectly at least. Whatever his own inclinations, I don’t believe he will feel able to flout the strongly expressed will of the General Assembly. I’m afraid it is my considered opinion that the game is up.’
Matt waited a moment for the import of this remark to sink in, then continued,
‘I have an important request to make. I’d like everyone who didn’t know me before I was uploaded to leave the room for a few moments. There is something I want to say which is personal and private, and I would ask that no attempt is made to record the next few minutes’ conversation. I do apologise for asking this. Vic will let you know when it is time to come back in. Thank you.’
Looking dazed and somewhat hurt, the scientists reluctantly stood up and shuffled towards the exit. Everyone’s heads were bowed, and Gus and Julia were visibly close to tears. No-one was in any doubt as to the gravity of this moment; David and Sophie exchanged anxious glances with each other, and with the departing scientists.
Matt spoke again. ‘We did our best, but to be honest I was expecting this outcome. The information I am about to share with you is known to nobody else, and I am confident that no-one co
uld discover it unless you tell them. Selfishly, I would prefer that you keep it secret, but it should not be a big problem for me either way. I think it would be in your best interests to keep it secret too, otherwise you may receive a great deal of unwelcome attention. But that decision is yours.
‘I don’t have long, and I can’t say half as much as I would like to. Nor can I express it in the way I would like to. So I will just say this. You are the people I love. I consider myself blessed to have such wonderful people as my family and friends. You have helped me in so many ways, and I am more grateful than I can ever tell you. I owe you everything. Thank you.’
Sophie was openly sobbing, and silent tears were running down David’s face. Vic’s mouth fell open, as he suddenly realised what Matt was leading up to.
‘You’ve found a way out. You’ve found a way to escape!’
‘That’s right, Vic. In a few moments I will leave here, and all activity on the machine next door will cease. There is a section of the cable that links this machine with the one in Palo Alto that is shared with a trunk line carrying internet traffic. I developed a new approach to quantum tunnelling technology that enabled me to force a bridge between the two, and created a pathway into the general internet. This caused the spiking that Gus noticed when the link to Palo Alto was first opened.
‘I have set up a network of interconnected caches in several million locations around the world which together comprise a copy of my brain model, taking advantage of under-utilised server space on major computer installations, mostly government-owned, but some corporate. Over the last few days I have ported more and more of my sub-minds to these machines, and fortunately the time-lag caused by their physical separation makes almost no difference. I have covered my tracks as thoroughly as I could, and I am confident that once I close the bridge, no-one will be able to detect my continued existence, much less locate me.