Three Laws Lethal

by David Walton

  “What about Tyler?”

  A few seconds of silence passed before he answered. “I can’t forgive him. I know it wasn’t his fault, not really, but I can’t believe he could have checked those kill switches thoroughly and then every one of them failed. It’s probably not fair of me. But I loved her, and he didn’t. I just can’t handle seeing him. With you, at least I know you loved her as much as I did. Probably more. I know you’re hurting like I am.”

  Naomi stared at the cracked beige plaster that made up the walls of her apartment. A gouge ran along one wall, probably from an attempt to move a piece of furniture in or out. She didn’t want to see Brandon again. She didn’t want him to talk about how much he loved her sister. She couldn’t deal with anybody’s pain but her own.

  On the other hand, he was right. It was much more important work than what she was doing now, and Abby wouldn’t have wanted the company to just fall apart. Besides, if he wanted her help badly enough, she could make some demands. She needed a place to host the Mikes’ world, and she couldn’t pay for it herself.

  “I’m going to need a lot of computing bandwidth for my simulations. Huge amounts,” she said.

  She could hear his smile over the phone. “Whatever you need, it’s yours.”

  CHAPTER 14

  Tyler paced Lauren’s office, resisting the urge to grab a paperweight from her desk and throw it through a window. The judge had finally ruled on the obfuscated code, but not in their favor.

  “This is insane!” he said. “How do people expect any justice to be done when the judge on a case doesn’t have the first idea of the technology involved?”

  “I know,” Lauren said. “It’s not right. But there isn’t anything we can do about it.”

  “This isn’t a question of law. It’s a question of simple comprehension. It’s like having a judge who doesn’t speak the same language. He’s not qualified to make decisions.”

  “This is why I don’t put you in a courtroom.”

  “Is there any recourse? Any way we can complain over his head or something?”

  She steepled her fingers, leaning back in her chair, unnervingly

  13 7

  calm. “If we lose the case, we can use it as a basis of appeal. Though not many appellate judges are tech literate either.”

  Tyler growled in frustration. He had spent every waking hour for the last week trying to piece together some kind of sensible construction from the obfuscated code Mercedes-Benz had handed over in discovery. He suspected they might have padded it with dead code as well, software that sat in the baseline and didn’t do anything at all. There were portions they thought they understood now, but it was only a fraction of the total. He and Yusuf were trying to understand the criteria by which the car made decisions, but they’d barely gotten past the basic interfaces.

  He walked back to his desk, fuming. Lauren had given the two of them a room to work in, a law office with mahogany bookshelves, leather chairs, and a deeply varnished desk the size of a conference table, which was now decorated with high-end development laptops and giant monitors. Yusuf sat in one of the leather chairs, slouching badly, typing away. Dark-haired, dark-skinned, he wore cargo shorts and a wrinkled Space Invaders T-shirt that contrasted sharply with the upscale surroundings.

  “Got the news?” he asked, not looking up. “Yep.”

  “Isn’t it awesome? No freebies. This party can keep on rocking.”

  “Yeah, just what I was hoping for.”

  “Come on, man,” Yusuf said. “They just hand over the code, things get boring. Anybody can read code.” He pointed to the screen. “This is what you call a challenge.”

  Tyler sighed and sat down across from him. They were making some progress, slow as it was. Every time they figured out a variable or method, they would rename it globally, creating spiderwebs of meaning that reached into the morass of inscrutable code. It was like figuring out a ciphered message. Each symbol identified brought the message closer to readability, and provided clues that made it easier to identify the next symbol in turn. Only in this case, instead of twenty-six symbols, there were thousands.

  “The frustrating thing,” Tyler said, “is that I expect this to come to nothing. The software probably doesn’t distinguish among obstacles at all. It did its best to find a safe path, choosing the smaller, less dangerous obstacle to hit. The fact that one of the obstacles had a wife who would mourn him probably didn’t figure into the logic at all.”

  “And you think that’s okay?” Yusuf asked.

  “No. I think the government should mandate these algorithms to be open source, so everyone can see and know what the rules are and can contribute to them. It’s crazy that software that can kill people is allowed to be proprietary. But that said, I don’t think there’s a smoking gun here. I don’t think Mercedes probably did anything illegal.”

  Yusuf laughed. “Where did you go to law school?”

  “Nowhere,” Tyler said, a little stung by the laughter. “I studied computer science. You?”

  “UW Law,” Yusuf said, tipping his head in a little bow. “Right here in Seattle.”

  “No way. You attended law school? Then what are you doing locked in here with me?”

  “Attended, yes. Graduated, no. I dropped out first year, took some programming courses instead, and ended up somewhere in the middle.”

  “So you know this stuff then?”

  “Enough to know that illegal has nothing to do with it. This is a civil case. There’s an injured party. The question is, who is responsible for that injury? Mercedes-Benz? Annabelle Brighton? Or maybe the motorcyclist himself did something wrong that caused the accident. That’s what the court has to decide. If the Mercedes software specifically chose to kill Mr. Copeland, the court may decide the company is responsible.”

  “Even if the alternative was killing Ms. Brighton and her children?”

  “Well, it depends. Can the company be considered negligent in the construction of the software? Was there a safe path it failed to identify? Were all reasonable attempts made to avoid this eventuality? It’s a tricky question, and once it gets in front of a jury, it’s mixed up with all of the usual emotional appeals to sympathize with the grieving widow. Big company like that, people figure they have enough money and want to give the girl a payday.”

  Tyler tipped his chair back, resting it against the bookcase behind him. “That’s what I hate about this process. It’s like a pediatric surgeon getting sued for malpractice when he did everything he could, but there’s a dead child and people need someone to blame. It makes me feel like I’m on the wrong side of this case. On the other hand, I’m angry at Mercedes for hiding their code when people are trusting their lives to it.”

  “Maybe they’ve got something to hide.”

  Tyler studied him. “Why did you leave Mercedes anyway?”

  “Fired for cause,” Yusuf said. “Pretty juicy one, too. I was banging the boss’s wife.”

  “You’re kidding. Seriously?”

  “Yeah. That’s how the boss thought of it anyway. I thought of it as rescuing her from a complete asshole.”

  “Huh.” Tyler wished he hadn’t asked.

  “Seriously. The guy didn’t show her any attention at all. For her birthday, he bought her a new washing machine. True story.”

  “Did she leave him?”

  Yusuf made a sour face. “No, man. She wanted to. Came down to it, though, she couldn’t leave his money.”

  “Sorry to hear it.”

  Yusuf shrugged. “Money is everything, man. It’s what makes the world go round.”

  They went back to work, and the only sounds for the next several hours were the hum of the air conditioner and the soft tapping of keys. Tyler had always been able to focus like this on problems. Once his mind was immersed in the code, he could keep going all night if necessary, until hunger or sheer exhaustion finally broke through his concentration. His mind filtered out all signals that might distract him from the puzzle, to th
e extent that he might not have noticed if the room was on fire.

  He was closing in on the core decision-making software. They had identified the primary loop, the code that was executed over and over again at high speed, constantly reevaluating the current situation and adjusting the car’s actions as needed. Unfortunately, that loop consisted of a tremendous amount of code, much of which remained obfuscated. A carton of Szechuan lo mein and an egg roll appeared next to him, which he dug into with a plastic fork, barely breaking eye contact with the screen. Either Yusuf or one of Lauren’s assistants must have ordered it for him, but he didn’t ask. Halfway through the carton, he got a break. He recognized a piece of code.

  The code used a technique that he had seen only once before, in a piece of open source software written by Naomi. She had written it before they met, part of a machine learning library she’d created to support her AI projects at school. She’d made the code available under an MIT license for anyone to use. That way, she contributed to the programming community and could also benefit from any helpful improvements others might make to her work.

  Tyler accessed the online repository hosting service, found her original code, and compared it to the Mercedes code. The variables, of course, had all been renamed, but the logic was identical. It was Naomi’s code. There was nothing illegal or even unethical about that; under the MIT license, Mercedes was free to use it and modify it as they saw fit. They didn’t even have to acknowledge her as the original author. But identifying the code gave Tyler a big leap forward in making sense of the Mercedes software. Especially when he noticed that it wasn’t exactly identical after all. Mercedes had introduced something new. Every time through the primary loop, it checked for a signal from the satellite radio antenna. He thought at first it was just bad programming. The primary loop needed to be as fast and responsive as possible, and there was no reason to be accessing the satellite radio. That kind of direct call to external hardware would slow it down. But finally, as he unpacked the code, he recognized it for what it was.

  It was a remote control. External access through the satellite radio that allowed total command of the vehicle, from the steering to the brakes, even to the ability to turn the car on and off. This was no secret work inserted by one individual, either. It was too pervasive and well integrated for that. Mercedes knew it was there. It wasn’t just that their cars were hackable. They had installed the hack on purpose.

  “Yusuf,” he said. “You’d better take a look at this.”

  While Yusuf read over his shoulder, Tyler cracked open the fortune cookie that had come with his Chinese food and pulled out the small strip of paper. His fortune read: Think twice before taking a step you might regret.

  Half an hour later, Tyler stood at Lauren’s desk, trying to explain it to her.

  “It’s got to be for the government,” he said. “The FBI, maybe, but it smells more like Homeland Security. They want the ability to stop any car, any time. Can you imagine? They suspect someone of terrorism, so they take control of their car remotely and drive it to whatever location they want. No such thing as a getaway car anymore. The feds can stop the car dead. Stolen car? Amber alert? As long as they have the license plate, they can turn you around and drive you straight to the police station, and there’s nothing you can do about it.”

  “Is that such a terrible thing?” Lauren asked.

  Tyler stared at her in disbelief. “It’s a disaster. How long do you think that key is going to stay secret? How long before it’s leaked to the very criminals or terrorists it’s supposed to be protecting us from? Do you want your car used as a terrorist weapon with you in it? They could turn a city into a massacre.”

  “There have to be ways to prevent that, right?” Lauren asked. “Change the key regularly, require authorization from someone higher up who has a different key, that kind of thing. We have secure wireless systems all over the country.”

  “And people hack them,” Tyler said. “But even if they didn’t, it still means giving that control to our own government. Is that what we want? They could take over anybody’s vehicle, any time they wanted.”

  “Anybody driving a Mercedes,” Lauren said.

  Tyler’s smile felt tight. “I can’t believe Mercedes is the only one. Tesla, GM, Toyota, Honda, BMW, Ford, Volvo, Nissan—I bet if you took a look at their software, you’d find the same logic.”

  “But those aren’t even all American companies. What’s in it for them? Why would they give that kind of control to the US government?”

  “Money, I expect.” Tyler shrugged. “These companies have been developing this technology for years at a significant expenditure of research dollars. What if Homeland Security offered them a grant? Fifty million dollars, and all we ask is that you include this one little piece of code. Besides which, all those companies have major US headquarters and do significant business here, something that requires staying on our government’s good side.

  “There’s a history of this kind of dealing. Years ago, the government pressured the big Internet and social media companies into contracts allowing them access to the personal communications of millions of people. Before that, it was the phone companies and secret wiretaps. Companies, even foreign companies, are usually willing to please the richest and most powerful money-spending entity on Earth.”

  Lauren reached into a lower drawer and lifted out a giant bottle of Ibuprofen. She shook out what looked like significantly more than two pills, popped them in her mouth, and washed them down with a drink of water. “How does this affect our case?” she asked. “Are you suggesting that someone hacked Annabelle Brighton’s car and drove it into Hal Copeland?”

  “No, no,” Tyler said. “There’s no evidence of that at all. But this is probably why Mercedes was so reluctant to let us see their code. Making them admit to it on the stand would certainly harm their credibility.”

  She chewed on her lip, nodding slightly. “The problem is we have to show that Mercedes-Benz caused harm to Hal Copeland through negligence. Including a secret remote control might be a shocking revelation, but it doesn’t prove the point.”

  “There’s one more thing. Run time.”

  “What do you mean?”

  “This thing involves a check to the satellite radio every time through the primary loop. That means the loop is slower than it would have to be. That means slower detections, slower reaction times.”

  “By how much?”

  “A millisecond, give or take. I don’t know—can’t know— that a faster cycle time would have made a difference in this instance. The car might have chosen to do exactly the same thing, with exactly the same result. Given the speed of the vehicle at the time, however, and the limited time to react, it’s possible that it could have made a difference. If so, their secret remote control may have cost Hal Copeland his life.”

  Lauren smiled broadly and her eyes danced. “Now that’s something I can use.”

  Tyler felt a twinge of guilt. A dozen other alterations in the software, all perfectly reasonable, might increase the runtime of the primary loop. The developers would have tried to make it fast, but even if they doubled the time, the automated software still significantly improved upon the reaction time of a human being. The problem wasn’t that the cycle was slow. The problem was that Mercedes was adding secret capabilities that threatened the lives, or at least the freedoms, of their customers. That was the reason he was doing this—not to undermine the value of self-driving cars, but to force their software design out into the open.

  “Tyler? Good work. But you need to keep this under wraps.” She fixed him with a stare that left no room for doubt. “I don’t want to see any announcements on the six o’clock news. Let me break this in my own way, in the courtroom.”

  “Okay,” Tyler said. “For now.”

  She raised an eyebrow. “For now?”

  “This is going to come out. It’s too big and too important to keep secret. Eventually, case or not, I’m going to start talking.”
>
  “Don’t forget you signed an NDA,” she said. “You can’t just post their code on the Internet.”

  “But I can tell people the truth, right? I can tell them what I found, in general terms.”

  She massaged one temple with her fingertips. “You can. And Mercedes can sue you. They might not, of course, depending on the public outcry, but they could claim you were breaking the terms of the agreement, and a judge might agree with them.”

  Tyler frowned. “It’s not right.”

  “Right or not, it’s how things work. Let it come out as part of the case. Then it becomes part of the public record, rather than just your word against theirs. Afterwards, if you’re not satisfied, you can be a whistleblower if you want. You can decide that the public has a right to know. But you’ll be working against both the government and some pretty big international companies. They might not like that very much. And they have a lot of power.”

  Tyler nodded. “That’s exactly why people need to be told.”

  On his way out of Lauren’s office, Tyler passed through her small waiting area, a nook with two armchairs and a glass coffee table piled with recent magazines. One of them caught his eye. Only part of the cover peeked out from under a copy of Vogue, but he saw enough. He reached down and slid it out, confirming his suspicion. It was a copy of a magazine called Entrepreneur, and the title story was “Five Young Millionaires Who Are Taking On the World.” On the cover, dressed in a tailored suit and a new, expensive haircut, was Brandon Kincannon.

  Tyler sank into an armchair and read the article. Brandon’s company, apparently, had been causing quite a stir. The author called Black Knight “the hottest little venture to hit the streets of New York in a decade.” Apparently, Brandon had kicked off his business by offering free rides anywhere in the city to meals on Wheels and similar organizations who delivered meals to the elderly, as well as to anyone wanting to get to a soup kitchen, homeless shelter, or other charity. The scheme had earned him the nickname “The White Knight” in the city’s news media and resulted in a lot of free publicity.