The image changed to that of a burned-out Starbucks on an urban street. Then another photo from a newspaper showing a burned-out SUV beneath the headline SENATOR ASSASSINATED IN TERROR BOMBING. “We need only consider the recent unsolved terror bombings here in the United States to recognize how critical visual intelligence is to our future.”
Strickland scanned the faces of his audience. They were with him.
“How do we imbue machines with this ability? We do this by emulating the way humans process spatiotemporal events. Human visual cognition is closely attuned to change, and it’s these changes that create what we call ‘attention states.’ We acquire ‘attention states’ from video imagery through an algorithmic mechanism that includes notions of focus of attention, markers placed on salient objects, and the critical relationships between those objects in terms of motion and contact. These are necessary to distinguish individual events from one another. A series of attentional states over time then becomes a visual attention trace—or VAT—which begins to form the elements of a story. One that can be programmatically narrated through machine-readable text—text that can then be algorithmically searched for relevance, in real time, by an ‘audience’ of other, simpler programs. This is why we call our system ‘Raconteur’—because it tells the story of what’s happening in a way that common systems can understand. And like any good storyteller, ‘Raconteur’ remembers how the current scene fits into the whole.”
Strickland knew that his combination of youth and poise would be an advantage here. Disruptive technology was like that. Now, at twenty-two, he was leading a team that was about to revolutionize visual image processing. Although he wasn’t the driving force behind the innovations, he did know how to spot and recruit talent to his work teams. If history was any guide, that was the primary skill necessary for success in Silicon Valley. Being able to spot a good idea and knowing who could make it work. Removing obstacles and inspiring others, that was the biggest part of innovation.
“We have worked with DARPA’s technical staff to coordinate the following demonstration, in strict adherence to the Mind’s-Eye Project guidelines. Please remember that our system has not been previously exposed to the images that you—and it—are about to see. We look forward to taking your questions after the test. Until then, ladies and gentlemen, I give you ‘Raconteur,’ the storyteller. . . .”
More light applause as the screen went black.
Strickland stepped aside as two smaller screens glowed to life up front—one bearing the title “TCTO Phase 1—Recognition Test.” The other screen displayed a blinking cursor.
Strickland moved to the side to stand with his project team, bracing for whatever came next. He cast a tense look at his development lead, Vijay Prakash, but the handsome, dour Bengali ignored Strickland’s arched eyebrows and looked to the screen. The rest of the grad student crew—Sourav Chatterjee, Gerhard Koepple, Wang Bao-Rong, and Nikolay Kasheyev—nodded in acknowledgment of the moment. Then they all turned to watch the screens too.
The words “TCTO Phase 1—Recognition Test” soon appeared also on the right-hand screen. The twin projections were set up so that whatever appeared in the left-hand screen, Raconteur would have to make sense of and describe in text on the right-hand screen.
Strickland felt relief wash over him as he stood in the darkness. Failing simple character recognition while reading the title card would have killed them, but then, OCR was handled by a licensed library, not their code. Still, he knew the DARPA judges wouldn’t cut them any slack for choosing a bad library.
But the test was already moving on. No time to ponder disaster scenarios. The left-hand screen changed to black-and-white surveillance video. It depicted a woman walking down an office hallway carrying a cardboard records box.
Strickland tensed again. He’d seen the VI algorithms work a hundred thousand times and had a pretty good idea how they functioned, but they’d never been run live in front of such an important audience. What happened next would decide the next several years of his life—of their lives—and quite possibly the trajectory of Strickland’s career. He focused on the blinking cursor on the right-hand screen—the Raconteur output panel.
As the video continued, text began to appear. . . .
Person carries object along corridor.
Murmurs of approval swept through the room, but Strickland remained tense. C’mon. Do it. Do it, baby. . . .
The cursor then began expanding on the details.
Woman carries box along corridor.
More murmurs and some clapping. Strickland cast a glance at the DARPA managers, who were nodding and talking softly among themselves. Taking notes. A wave of relief flowed through him. He’d had no idea how clenched he was, but now that initial impressions were good, the judges would be more receptive if there was a later glitch. He told himself that no matter what happened from here on, they had at least avoided a meltdown. They had gotten on the scoreboard.
The scene changed to an exterior; an American soldier standing on a littered street in some Middle Eastern slum, weapon slung and motioning to unseen people. A small—possibly Iraqi—child entered the frame behind him. Strickland felt the dread returning, as the text scrolled. . . .
Armed person . . . approached by child.
More applause and some actual shouts of excitement.
Strickland felt a smile crease his face before he clamped down on it. Too early to celebrate.
Uniformed soldier approached by child in street.
The hoots continued. So far so good, but Strickland knew the difficulty levels were only going to increase. As he watched, the system mistook another soldier entering the frame as a possible threat—#ALERT—armed person. Not too far off the truth, though.
The control frame faded to black and displayed the title: “TCTO Phase 1—Interpolation Test.”
Here we go. The complexity of visual concepts ramped up fast. It was why their system focused on deriving context first while interpreting a scene, and why it never forgot what it had seen previously. That was key to avoiding a lot of useless processing. Humans walking down a city sidewalk, for example, do not suddenly expect to see a mountain vista or a rolling sea all around them. That would be impossible—thus, even if these things appeared, they were likely to be graphical representations like ads, not the actual thing. Daisy-chaining events made it possible to take the known and use it as a base camp from which to explore the unknown—pushing that frontier back just a little at a time, like ants exploring terrain.
As Strickland knew, even a person with Down syndrome was a generalized genius compared to special-purpose computer algorithms. Breaking things down to their simplest elements was the only way to accomplish anything useful. Prakash had worked out the architecture, and the design made Strickland’s head hurt. But if the damned thing worked, he’d forgive all of the man’s arrogance.
The scene on the left changed to a woman in a burka—a burka! What U.S. troops called a “BMO,” short for “black moving object.” DARPA bastards. No face, no clear view of her arms or torso. On-screen she resembled a walking bag. But if memory served, Vijay and Gerhard’s gait detection code should help assign the attribute of “human” to walking objects—and along with “humanity” came implied geometry, potential actions, and patterns of movement. The burka woman was moving along a narrow village road carrying what appeared to be a plastic water jug on her head.
The room waited with bated breath. Then the text started scrolling.
Person carries object down street.
Okay, so far so good.
The woman entered a dwelling through a doorway on the left, and the system correctly described her disappearance. Then all was quiet for a moment, until she reemerged without the jug on her head. This was the real test. Cognition.
#ALERT—DROPPED—ITEM: Person observed carrying object into building and leaving without it.
Strickland felt the importance of this moment as loud applause filled the room. They had just pas
sed the bomber test. Years of work flashed before him. He felt the backslaps of his teammates, and he turned to their smiling faces in the semidarkness. He even grabbed hands and side-hugged Prakash. They’d never gotten along well—always struggling for the reins. But this moment was what they’d been working for. Even the eternally serious Prakash gave the barest hint of a smile. A smirk, really.
Strickland had to admit the guy knew what he was doing. “Great work, Vijay.”
Prakash nodded. “It’s a start.”
Prick. Couldn’t he enjoy anything?
There were calls for quiet as the test continued, but a warm tingling had settled across Strickland. They would get their research grant. He knew it now. The excited discussions among the judges told him they’d outperformed anything they’d ever seen. His professional career had begun, and he would forever remember this moment. He couldn’t wait to tell Sandra.
But then he remembered that they weren’t seeing each other anymore.
* * *
Strickland popped the cork on a bottle of cheap champagne and let foam spew in all directions as his research mates screamed in jubilation. Back in the KSL lab cluster on the second floor, there was much to celebrate. The lab was an open workspace with HD digital video cameras clamped to brackets and on tripods scattered here and there, rack servers in one corner, their LED lights flickering as though in time to the music. LCD monitors on desks and mounted to the ceiling scrolled Raconteur-generated text of the festivities . . . most of it not too far off—but then they would now have a federal grant to perfect it, wouldn’t they?
What they’d believed to be groundbreaking work had been recognized. The venture capital arm of a U.S. intelligence agency had tentatively agreed to finance their research project, but along with that came top-level introductions to other private venture firms. His team now represented the very bleeding edge in the field of visual intelligence. All their personal rancor and disagreements had been for a purpose, and now the entire crew shouted another toast, enjoying the moment—Chatterjee, Koepple, Prakash, Wang, Kasheyev—a truly international team. And then there were the other lab teams in the cluster, along with their faculty advisors. There were spouses and significant others, as well—turning it into a full-scale party. Strickland wished he had someone to share it with too. But that would come in time, especially now that success had found him. In a few years he hoped to be a partner in some venture capital firm on Sand Hill Road. He was on his way.
Strickland stepped up onto an office chair as someone steadied it for him. He raised his glass and Lei Li, their mentor, called out for quiet—with everyone suddenly shouting, “Speech! Speech!”
In a few moments the music got turned down, and in the sudden silence Strickland raised a plastic cup sloshing with champagne. “Guys, I want you to know what an honor it’s been to work with this team. I’m aware that the brilliance lies not with me, but with all of you”—he pointed—“Sourav, Gerhard, Bao, Nik, and of course, the inimitable Vijay.” There was applause and cheers with each name.
Prakash stood near the wall, arms crossed, watching Strickland with something approaching disdain. Prakash wore his trademark khaki pants and blue Oxford button-down shirt. His hair, as always, closely cut and perfect. Bollywood good looks.
What was the deal with this guy? Why couldn’t he just loosen up? Strickland tried to ignore him. “I think it’s fitting that in a building where Google was born—bringing Web search to the world—visual intelligence would also be born, bringing the ability to search reality in real time. You guys will make history, and I’m just glad to be along for the ride.”
“Hear, hear!” Another cheer went up and glasses were drained. The music came up again, and the crowd resumed shouted conversations, while lab members mugged before the video cameras.
Strickland noticed Prakash still staring daggers at him, so he waded through the crowd to him. “Vijay, what’s wrong with you, man? You look like Kolkata just lost the cricket finals or something.”
Prakash eyed him. “Speaking as one of the ‘brilliant’ team members, I don’t appreciate you monopolizing the conversation with DARPA. We make decisions as a team, Josh. The only reason you have the title of ‘team lead’ is because you’re good at dealing with suits, and it keeps you out of my way.”
“I was just setting a date for the next meeting. We’ll confirm it by e-mail later.”
“We need to be involved in every decision that occurs—no matter how small.”
“We’re not a hive mind, Vijay. Occasionally tasks need to be delegated, and I don’t see the point of bothering you with—”
Prakash got in his face, finger jabbing. “This is not the first time I’ve had to remind you. I do not work for you, Josh, and I expect you to represent the interests of the team, not just yourself.”
“Whoa, whoa! Hold on a second. We all agreed from the get-go that since I’m the extrovert that it would be best if I did the talking, especially with the defense folks. That’s what I’ve been doing.” He gestured back to the chair he’d been standing on. “Did I not just lay literally all of the credit onto everyone else’s shoulders?”
“And rightly so. The next time we communicate with the U.S. government, I expect to be copied, Josh.”
He studied Prakash’s face. Did he know? There had indeed been a few other communications, but it had only been an oversight. “Look, I don’t know what you think I’m up to, but in case you haven’t noticed, you’re sort of critical to this project. I can’t screw you without screwing myself. You’re on the patent applications.”
Strickland wondered if Prakash’s highly competitive upbringing was behind this. He knew that the guy’s father was a real type-A businessman—a real ballbuster who practically rode his sons with a riding crop. Old school. Very class conscious too.
Was that it? Strickland sometimes wondered if Prakash thought less of him simply because Strickland was straight middle class—the son of schoolteachers. He’d seen photos of the Prakash summerhouse in the south of France, and photos of Prakash literally in a polo outfit, holding the reins of a pony—like he was friends with the British royals or something. It was easy to forget that this guy might just look at him like he was one of the help.
The more he thought about it, the more it irritated him. He raised his cup again to Prakash and gave him an obnoxious wink, aiming a finger pistol at him in oily salesman fashion. “Who loves ya, guy? Eh? Who loves ya!”
Someone came up from behind and ruffled Prakash’s hair, laughing. Strickland took the opportunity to step away, heading outside for some air. He worked the room on the way out, and by the time he made it to one of the hallway doors, he could see the light on in one of the tiny, windowless offices that surrounded the lab cluster. There sat the childlike form of Nikolay Kasheyev, their visual processing expert, sitting at his desk.
Kasheyev had some sort of pituitary condition that gave him the appearance of a twelve-year-old, even though he was in his mid-twenties. He was often told he was a child prodigy, and every time people made that mistake, Strickland winced on his behalf. But the Russian seemed used to it.
He could see Kasheyev examining multiplexed video streams, tiled into squares on his screen. Strickland rapped his knuckles on the office doorframe. “Hey, Nik, why aren’t you joining the party, man?”
Kasheyev spoke without turning around. “The ravens are back.”
“What ravens?”
Kasheyev finally bothered to turn around. “You never listen to me, do you?”
Strickland was too busy wondering how much Kasheyev’s eyeglasses cost. It seemed that among the team members, only Strickland had come from humble circumstances. Everyone else seemed to have parents with money. No one else had had to work through their undergraduate years. At least not at menial jobs. It seemed like Kasheyev wasn’t even aware that they’d probably laid the groundwork for becoming wealthy men today. It was as though it barely mattered to him.
“Josh.”
“Ah. Of course I listen to you. I just don’t retain what you say.”
Kasheyev pointed to an array of six video images tiled on his screen—surveillance cameras that Strickland knew were being “watched” at all times by Raconteur algorithms. They’d had the system running more or less continually (in progressive versions) for almost two years, watching the public spaces all around the Gates building. Watching the comings and goings of students, automobiles, everything. That was part of the power of this system: persistent surveillance. Watching over time. Noticing patterns and changes to those patterns. Continuing to deduce meaning from what was seen, storing the symbolic representations, and sending alerts—in this case to Kasheyev’s iPhone—whenever anomalies were detected. Specific alerts could be created, and apparently Kasheyev had started fixating on these birds. It was the least of his eccentricities.
Kasheyev tapped at one of the images, where what appeared to be a black raven gazed at their offices from a tree branch. Clearly the video was running, because Strickland could see the leaves of the tree swaying in a slight breeze.
Strickland leaned in to examine the screen, and then examined the Raconteur log, highlighted for the entry. Raven perched in tree.
“Okay, Raconteur recognized a species of bird. We’re amazing.”
“That’s because it searched tagged Web images for comparisons—object identification is not really my point. It’s that there have never been ravens in these trees. And, look, there’s another raven here, up near the Packard Building to the south. . . .” He clicked around the screen and brought up a larger image of another raven perched on the parapet of the architecturally modern electrical engineering building across the street. He scrolled backward in time, and Strickland could see that as the sun rose and set, the twin, but lone, ravens returned day after day—but then one day, they weren’t there. “It started a week ago. And they’ve come back every day since.”
Kill Decision Page 3