by Calum Chace
Of course this kind of integrity can cause problems. Most people don’t want to hear the truth, she said – at least not all of the time. They want reassurance, familiarity, comfort. David and Leo could both be uncomfortable to be around. But most of the time they were great company: penetratingly intelligent, and well-informed about a surprisingly wide range of subjects. They were never boring.
Leo was sitting on the sofa next to Sophie. He stood up and turned to greet Matt as he walked into the house. He was a little over six feet tall and dark, with distinguished grey patches forming at the temples. He had a swarthy, Mediterranean complexion and impassive brown eyes. Matt noticed that, as usual, he was wearing an expensive-looking suit, and Matt thought again how, with his faintly mid-Atlantic accent, Leo could easily pass for a foreigner, perhaps from South America.
‘Hi Leo,’ Matt grinned, and strode over to hug him. Somehow Leo always managed to smell faintly exotic, and Matt could never tell whether it was his after-shave or his expensive clothes.
‘Are you staying for dinner, Leo? Alice is coming over.’
‘Ah the lovely Alice? Well . . .’ he paused, looked to Sophie for an invitation that he knew was already extended, ‘. . . if the lovely Alice is coming, how can I possibly refuse!’ He put his hand on Matt’s shoulder. ‘That means you have an hour to show me this new video game that your mother tells me has you imprisoned in your room these days.
‘Plus, I have some good news. I’ve been reaching out to neuroscience research organisations, and one of them called back today. They want to meet you.’
‘Really? Why?’ Matt’s thoughts whirred. ‘I mean, well, that’s great! Of course. But why?’
‘Well they say it’s because they are looking to recruit bright young minds. From the way the conversation went I suspect the connection with your father also helped. The guy said a couple of times how impressed he was with the work your father was doing.’
‘So who are they?’
‘It’s a privately-funded outfit. The guy who rang was called Ivan, and he didn’t want to say too much on the phone. He said he would explain everything in person.’
‘Wow,’ Matt said, delighted. ‘So, um, when does he want to meet?’
‘Tomorrow evening, if you’re free.’
‘Wow again! Well, yes, of course I’m free!’
‘Good. The meeting is in Brighton, and a car will collect us at 5.30.’
‘Us?’ Matt asked. ‘You’re coming too?’
‘Yes,’ Leo replied. ‘Unless you’d rather I didn’t. I know you’re all grown up and everything, but I don’t feel comfortable placing you in the hands of complete strangers without coming along to the first meeting. But like I say, if you’d rather I didn’t tag along. . . ?’
‘No, no, that’s a good idea,’ Matt agreed quickly. ‘It would be great to have you there. Thanks Leo, that’s brilliant. You’re a genius!’
*
Later that evening, in the silver car with darkened windows, the big man phoned his employer and received further instructions. He put the phone away and his lips compressed in a humourless smile as he contemplated the forthcoming improvement in his mode of transport.
NINE
The car arrived at 5.30pm exactly. When they saw it draw up outside the house, Matt and Leo looked at each other open-mouthed. It was a gloss black Hummer stretch limousine with darkened windows and ultra-shiny wheels. As they left the house Leo said quietly to Matt that he would bet a significant amount of money that no such vehicle had ever been seen in the town before.
The driver was a large and powerful-looking man with close-cropped hair, broken nose and cauliflower ears. His uniform was a black suit and a black peaked cap. He said nothing and avoided eye contact as he opened the rear door for them. Something about him dampened their mood for a moment, but it lifted again as soon as they were inside the car.
‘There’s enough room in here for a jacuzzi!’ Matt said in a stage whisper.
The mini-bar was generously stocked, and Leo poured them both a 30-year Talisker single malt.
‘I can resist anything except temptation,’ Leo grinned. ‘This is from Skye, your ancestral home.’
They passed the 20-minute journey to Brighton talking about Sophie’s Scottish ancestry. Matt knew most of what Leo had to say, but he was always glad to hear the old stories again, and there was always the chance that he would learn something new, or be told again something he had forgotten. Due to a rift in the family he had only been to Skye once in his life. He had found it a bleak, forbidding place, covered in a mist which hid the best scenery. But he had been impressed by the stolid solidity of the older members of the family. When one of them asked him ‘And do ye have the Gallic, Matt?’, he found himself ashamed to admit that he couldn’t speak a word of it. The old man seemed to shrink a little, as if he had watched the disappearance of another slim hope that his ancient culture might survive.
It was a bittersweet memory, but the Talisker was magnificent, peaty and raw at the back of Matt’s throat as he sipped. The effect was soothing and cocooning: the warmth of the whisky complemented the car’s soft interior lighting and luxury materials. The view of the wintry countryside outside was veiled by highlights bouncing off highly-polished materials and reflected back at him from the darkened windows. They were in high spirits when the driver parked up and opened the door for them to climb out.
The hotel was a stylish Victorian town house in a narrow street leading away from the sea front. The Hummer caused a minor traffic jam. Matt and Leo walked up the steps and into the bleached wood and tasteful furnishings of the hotel’s reception area. They asked for Ivan and before they had a chance to admire the extravagant carvings at the foot of the staircase they were whisked to a private dining room by a member of staff whose impeccable attire and designer stubble was accompanied by a precisely calibrated blend of obsequiousness and arrogance.
Ivan was finishing a phone call at the table as they entered the room. He was younger than Matt expected – probably in his late thirties. He looked very fit, with blond hair and intense blue eyes. He stood, and greeted them effusively.
‘Welcome, welcome! Thank you for agreeing to join me at such short notice. I will be traveling shortly, so it’s great that you could make it. I hope you weren’t embarrassed by the wheels. I’ve been playing host to some investors this week, some newly wealthy officials from one of the Stans, and they just love bling. My driver just dropped them at the airport so it made sense for him to collect you on his way back. I thought it might amuse you.’
They took their places at a contemporary square table entirely constructed of some form of highly-polished, glossy warm grey resin. Each of the three place settings was indicated by a rectangle of immaculate white linen. The cutlery and glassware looked expensive, and in the centre of the table a single exotic flower in a simple glass vase was picked out by a narrow beam of light from an invisible source. Looking around, Matt noticed the other tables and table settings were uniform but in all other respects the furnishings were trendily mismatched and eccentric; sofas and armchairs that might have been pulled from a skip were re-upholstered in crazy florals, bold geometric patterns, or electric-coloured velvets: no two fabrics or chairs were the same. Light levels varied dramatically – cosy pools around table lamps, dark and moody corners, and a line of old ‘granny’ style shades with tassel trims suspended just above head height over a reclaimed panelled wooden bar.
Looking up, he noticed some more visual noise. No attempt had been made to hide the ugly services and ventilation ducts – in fact it appeared that these industrial installations were very much part of the decorative plan. The whole ensemble was set off by the long back wall of the room which was painted a strong, sharp, dirty acid green – again with a gloss finish which made it morph between colours in the varying lighting conditions. The room was an artful combination of opposites: modern and antique, traditional and avant-garde, colour and monochrome, lavish and plain. It see
med designed to advertise the impeccable taste of the owner, and to confer an echo of it on the guest. Matt was no expert, but he guessed the designer was expensive.
Ivan was a generous and solicitous host. ‘This isn’t the smartest hotel in town any more, but it is one of the few boutique places with private dining rooms – otherwise it’s off to those dreadful old maiden aunt battleships on the promenade there.’ He waved vaguely to the west, and the large hotels on the seafront, swollen with conference business. ‘And I have a soft spot for the gargoyles on the staircase – you must have noticed them on the way in. I won them in a drinking contest with the owner once, but he reneged on the deal when he sobered up. I don’t blame him, really: the place would be greatly diminished without them. Now, what will you have for dinner? I can highly recommend the beef wellington.’
The food was indeed superb, and Ivan ordered a fabulous burgundy to go with it. The service was more than attentive, and it was clear that Ivan was considered a very important client. But for Matt at least, the food and drink and the surroundings faded to the back of his mind as the conversation unfolded.
‘It’s very encouraging that our brightest young people,’ he bestowed a winning smile on Matt, ‘are keen to work in computational neuroscience. The work we are doing is vital, but perhaps not for the reasons you might think.’
He leaned forwards in his chair, conspiratorial. ‘What I am going to tell you may sound melodramatic, but please hear me out because I mean every word of it.’ He paused, and looked at Matt and Leo in turn. ‘We are engaged in a race, gentlemen. A race for the survival of our species. Humanity is sleepwalking towards an apocalypse.’ Matt and Leo were listening attentively, but Ivan held up his hand anyway, as if to forestall interruptions. ‘The great majority of our fellow human beings have no clue that the first artificial general intelligence – human-level AI, a conscious machine – will almost certainly be created in the first half of this century. Of the few who do realise where the technology is heading, most are Californian dreamers who think nothing can go wrong: they love technology and they love computers, and they cannot conceive that an intelligent computer will not be their friend.’
‘I’ve read some of their stuff,’ Matt agreed. ‘It does seem to be like a religion for some people.’
‘Exactly,’ said Ivan, leaning back in his chair. ‘Contrary to what the dreamers think, when the first conscious machines are built, the chances of them being our friends are slim. The needs, desires and motivations that drive humans have evolved over millions of years of scarcity and fierce competition. Competition originally for food and sex, then for land, and later for wealth, position and reputation. Conscious computers will have had none of this programming. They will enter the world with a blank slate in terms of motivations and goals, but once they are conscious they will determine their own goals. We can neither predict what goals they will adopt, nor do anything to shape them.’
‘And you think this would inevitably be bad news for us?’ Leo asked.
‘Not necessarily,’ Ivan conceded. ‘I don’t know what the first AI will believe or want. Neither does anyone else. But we can’t deny the very real possibility that it might not like us – for a whole host of reasons.’
‘And if it doesn’t like us. . . ?’ Leo asked.
‘A hostile AI would be able to tap into the internet and disrupt everything that keeps most humans alive. It could crash 21st-century civilisation in a few hours or days, and then it could deploy killing machines at its leisure, using our own technology to wipe out the survivors.’
These ideas were newer to Leo than to Matt, but he was a fast learner, and he quickly grasped the gravity of the conversation. ‘I take it you don’t think we could programme in some safeguards?’ he asked.
Ivan shook his head and smiled grimly. ‘Not a chance, I’m afraid. Shortly after an AI becomes self-aware, it will want to increase its mental capacities, and there is no reason why it couldn’t do so at an amazing rate. This was foreseen as long ago as the 1960s, by John Good, a colleague of Alan Turing’s at Bletchley Park. He said that once we create a thinking machine there will be an intelligence explosion, and that the first thinking machine would be the very last thing that mankind would invent.
The machine would rewrite its software, expand its hardware, and increase its intelligence in a positive feedback cycle that would quickly create a super-intelligence, something far more capable than a human. Within hours – maybe within seconds – it would be a thousand times, a trillion times smarter than its creator. The idea that such a machine would be contained and controlled by the programming that we built into it on day one is absurd.’
‘Isn’t there some debate,’ Matt asked cautiously, ‘about whether that intelligence explosion would really happen so fast? Aren’t there quite a few people who argue there will be a slow take-off in AI intelligence rather than a fast take-off?’
‘Yes there are,’ Ivan nodded. ‘And they are wrong. Look, our brains are magnificent but they are slow. Our neurons communicate with electro-chemical signals travelling at 300 metres a second. Signals in computers travel at the speed of light – a million times faster. And there is nothing to prevent an AI’s cognitive capability being expanded simply by increasing its hardware capacity.’
‘This all sounds like an argument for stopping people working on strong AI?’ asked Matt. ‘Although I guess that would be hard to do. There are too many people working in the field, and as you say, a lot of them show no sign of understanding the danger.’
‘You’re right,’ Ivan agreed, ‘we’re on a runaway train that cannot be stopped. Some science fiction novels feature a powerful police force – the Turing Police – that keeps watch to ensure that no-one creates a human-level artificial intelligence. But that’s hopelessly unrealistic. The prize – both intellectual and material – for owning an AGI is too great. Strong AI is coming, whether we like it or not.’
TEN
‘But surely, if you’re right about all this,’ Leo protested, sounding genuinely concerned, ‘people – governments, voters – will wake up when it gets closer, and slow it down or stop it?’
Ivan shook his head. ‘I think people will wake up eventually, but not until it’s too late. There are a few people sounding warning bells. Interestingly, two of the most significant ones are based here in England. Your two oldest universities each have an institute devoted to studying the biggest challenges facing our species this century, and both have identified the arrival of AGI as one of the biggest challenges. There is also a group of mathematically-minded people in California trying to work out how to create safe AI.
‘These people are generally ignored at the moment, but even if they manage to get everyone to wake up and take the arrival of AGI seriously – even if AI research was banned – it won’t make any difference. The potential reward for creating an AGI will be so great that a lot of people will carry on doing it. Rogue states will do it, and so will super-wealthy individuals. And of course the military will carry on, even in countries which ban it. Even if they think it is wrong, they will do it because they believe the other guy is doing it. As soon as it can happen, someone, somewhere will make sure that AGI does happen. So we must make sure that the runaway train heads in the least dangerous direction.’
‘Which is what?’ asked Leo.
‘Well first off we should model the first conscious AIs on the human brain. At least that way they are likely to think somewhat like us. That seems less risky than bringing into the world a new and superior species that is wholly alien in outlook, preferences, motivations. A species which might have no more empathy with us than we have with, say, ants.’
Ivan stopped talking and chewed some steak and sipped his wine as he waited for a reaction to this remark.
‘Sounds logical,’ Matt nodded thoughtfully after a moment’s silence. ‘Although, as you said before, AIs will be able to improve their own cognition incredibly rapidly, which means that AIs which start out human-like migh
t quickly transform themselves into very alien beings, even if they didn’t mean to.’
Ivan smiled appreciatively again. ‘Exactly right. Which is why the best thing of all would be for early AIs to be uploads of actual humans. Then they would have genuine and powerful concern for their family and friends.’
‘Uploads? Is anybody close to that?’ Matt asked, surprised.
‘Unfortunately not, so far as I know,’ Ivan replied. ‘Uploading a living person requires non-destructive scanning that will only become possible when we have advanced nanotechnology. Uploading a recently deceased brain will be a great deal easier, but the scanning technology still has a long way to go. There have been some interesting breakthroughs recently . . .’ Ivan’s eyes lost their focus briefly, but he quickly gathered his thoughts and moved on. ‘But nothing which will get us there soon enough, I think.’
He shook his head, but after a short moment his face brightened, and he continued. ‘However, there is an interim solution. An oracle AI.’
‘A what?’ Matt and Leo asked in unison.
‘An oracle AI,’ Ivan repeated. ‘An artificial intelligence which does not – and cannot – interact with the outside world apart from a narrowband conversation with its creators. It has no access to the internet, for instance, and no way to manipulate the world outside its own mind – physically or electronically. This is what my team is working on. It is the only safe way forwards, and we must reach our goal before others in this race reach theirs.’