Defying Reality

by David M. Ewalt

  But for all its success, Pokémon Go was a pretty limited example of augmented reality in action: the app could make a Pikachu appear in your living room, but the creature didn’t know it was there.

  * * *

  —

  That’s where Magic Leap came in. CEO Rony Abovitz described the company’s technology as mixed reality—digital content that interacts with the physical world. If augmented reality can make a Pikachu appear in your living room, mixed reality makes it come to life. An MR Pokémon could recognize its environment and interact with real-world objects; it might escape capture by ducking behind your couch, or even by turning off the lights so it can hide in the dark.

  “One of the reasons we call it mixed reality is because it’s taking the analog world as you know it and then blending in the digital all around you,” Abovitz said. “It’s a very different experience from chasing Pokémon on a phone, a very different experience from a VR headset. It should feel very natural and shouldn’t feel confined; you shouldn’t feel claustrophobic.”

  How does it do it? The centerpiece of Magic Leap’s technology was a head-mounted display, but Abovitz told me the final product would fit into a pair of spectacles. When I was wearing the prototype, an array of optics sat in front of my eyes, but didn’t block my view of the world; instead of drawing images on an opaque LED display, Magic Leap’s hardware projected an image using an optics system built into a piece of semitransparent glass.

  This photonic lightfield chip was Magic Leap’s most proprietary and secret technology, so the company wouldn’t say exactly how it works. But the basic idea is kind of like an old-fashioned TV. In a cathode ray tube, a stream of electrons scan across coated glass, excite phosphors that emit light, and draw a picture on the screen. In Magic Leap’s system, a stream of photons scan across the lightfield chip, bounce off nanoscale structures that reflect light, and draw a picture directly on your retina.

  “The idea is that we are simulating a signal that your brain has evolved to process,” Abovitz said. “What Magic Leap is doing is trying to get out of the way.” Instead of creating an image on a screen, the photon chip feeds your eyes information and lets your brain create the picture, he explained.

  The result is incredibly realistic virtual images. Since the photonic chip sends light directly into a viewer’s eye, it can display images at a much higher resolution than a physical screen can; it isn’t limited by the size of components like light-emitting diodes. The chip also eliminates many of the comfort issues associated with 3-D displays. With Magic Leap’s system, your eyes focus on the point where an object appears to be, not on the screen where it’s actually displayed. That makes 3-D simulations less taxing on your eyes—and more convincing to your brain.

  But there’s much more to this technology than an advanced 3-D display. Mixed reality is interactive, so the headset is constantly gathering information about the world around it, scanning the room for obstacles, listening for voices, tracking eye movements, and watching hands.

  In one of its demos, the Magic Leap team showed off a computer-generated “virtual interactive human,” life-size and surprisingly realistic. Abovitz and his team imagined virtual people (or animals or anything else) as digital assistants—think Siri on steroids, except with a physical presence that makes her easier to work with and harder to ignore. Ask your virtual assistant to deliver a message to a coworker and it might walk out of your office, reappear beside your colleague’s desk via his or her own MR headset, and deliver the message in person.

  In a mixed reality world, computing power isn’t confined to a gadget on your desk. It’s something that you can link to any object, real or virtual, giving it awareness of its location, intelligence about its purpose, and insight on how you might want to use it. “Think of it as the future state of computing,” Abovitz said, “where the world is your desktop.” First we had mainframes, then PCs, then mobile devices. If Magic Leap had its way, the next generation would be virtual.

  “This is not about entertainment or just playing video games,” said Thomas Tull, the billionaire founder of Legendary Entertainment. “This is a different way of interacting with the world, a new generation of computers. I think this will end up being a very, very important company.”

  * * *

  —

  Rony Abovitz had always surrounded himself with visions of a high-tech future. Born in 1971 to Israeli immigrants living in Cleveland, he grew up fascinated with computers and science fiction. “My generation are the children of Steve Jobs and George Lucas,” he said. “We grew up on that, and our brains all got scrambled . . . My friends and I all wanted to be Luke Skywalker and defeat the Death Star and build C-3PO.”

  When he was eleven, the family moved to South Florida; he started high school at thirteen, a year early. After graduation, he was accepted to MIT but chose the University of Miami, staying close to home. He received a bachelor’s degree in mechanical engineering in 1994 and a master’s in biomedical engineering two years later—and then he started thinking about Star Wars again.

  Abovitz cofounded his first company, Z-KAT, in 1997. “I decided to build the medical droid from Star Wars because I thought—I mean, this was the genuine impulse—I couldn’t build an X-wing fighter, because that wasn’t something I could explain to my parents after graduating,” he said. In 2004, Abovitz and several of his cofounders spun out Z-KAT’s robotics group into a new company, Mako Surgical, which made robotic arms to assist doctors in orthopedic surgeries. Demand for the droids was high, and in 2008 the company went public, raising $51 million.

  Working full-time at Mako and married with a young daughter, Abovitz found an outlet for his creative side in a project he called Hour Blue—his own fictional fantasy world, an alien planet full of fantastic creatures like talking robots and flying whales. In 2010, he started a new company, Magic Leap Studios, to develop the project as a graphic novel series and a feature film franchise.

  “I was the only employee, and it was literally in my garage,” Abovitz said. “My mom made a piece of canvas with some colored letters on it that said, ‘Magic Leap Studios.’” For help on the project, he used some of the cash he’d banked from Mako to hire Weta Workshop, the New Zealand–based special effects and creative-development shop best known for its work on The Lord of the Rings film trilogy, to develop imagery based on his story ideas and flesh out the world. Meanwhile, inspired by sci-fi novels like William Gibson’s Neuromancer and Vernor Vinge’s Rainbows End, Abovitz became frustrated that the augmented and virtual reality he’d read about in fiction wasn’t available in the real world and began thinking about how to make it so.

  “It was a unique moment . . . Reality and science fiction began to merge,” said Richard Taylor, CEO of Weta Workshop and a member of Magic Leap’s board. “The fictitious technologies that we were creating for Hour Blue were developing in parallel to the real-world applications of augmented technology that Rony was beginning to explore.”

  In 2011, Magic Leap Studios changed focus and became Magic Leap, Inc., with Abovitz hiring a small team to help him develop this idea of mixed reality. Before long, the company had working prototypes.

  “The first time we had a single pixel in space, where we could move it around the room, we were so excited,” Abovitz said. “Other people were like, ‘What is that? That’s just a dot.’ But we knew. I knew at that point this was going to work.”

  He also knew he was going to need a lot more money. Abovitz had initially funded the company with his proceeds from Mako’s IPO. After Mako was acquired in 2013 by medical device manufacturer Stryker Corporation for $1.7 billion, he invested some of those proceeds as well. Abovitz wouldn’t reveal the exact amount he spent to get the company running (“in the millions” was all he’d say), but he knew it wouldn’t be nearly enough.

  Fortunately, the technology sold itself. “When we’d talk to people about what we were doing, they didn’t bel
ieve us,” Abovitz said. “Then they’d fly in and go, ‘Oh . . . you’ve actually made these things happen.’ That was the dynamic of everyone who invested; [they went from] ‘This is impossible’ to ‘We want in.’”

  In February 2014, Magic Leap announced it had raised more than $50 million in seed funding from private investors. Eight months later, the company closed a Google-led $542 million Series B funding round.

  “We invested in Magic Leap because we believe their lightfield technology is the next big inflection point in technology after the PC, the Web, and the smartphone,” said James Joaquin, cofounder of Obvious Ventures, a San Francisco–based venture capital firm. “It has the potential to transform multiple global economic categories, including entertainment, education, and productivity.”

  Magic Leap’s record-breaking $794 million Series C round, announced in February 2016, was led by Chinese technology conglomerate Alibaba and included follow-on investments from Google and Qualcomm Ventures, plus new money from Fidelity Investments, JPMorgan, Morgan Stanley, and T. Rowe Price. “I don’t think it would be mischaracterizing it to say there was a frenzy,” said Thomas Tull. “This company has had absolutely no problem attracting blue-chip investors, and it’s extraordinarily well capitalized.”

  * * *

  —

  Rony Abovitz didn’t seem like the captain-of-industry type—unless your model mogul was Willy Wonka. Abovitz lit up like Roald Dahl’s genius chocolatier as he led me on a tour through Magic Leap’s new headquarters in Plantation, Florida, a campus more befitting his vision than the nondescript offices fifteen minutes away in Dania Beach. He pointed out “cool” machines, admired tools, and encouraged me to climb a ladder so I could check out high-end air filters in the ductwork. He was friendly and cheery, casual in speech and attire. On the few days I visited the company, his employees commented on what a nice guy he was as often as they mentioned his intellect.

  He was also liable to get lost in his work. On one Friday afternoon, he was missing in action a half hour after a tour of the new headquarters was scheduled to start—a not uncommon delay but a problematic one, since Abovitz comes from an Orthodox Jewish background and was planning to leave work early to observe the Sabbath. Eventually one of his executives discovered he had been on-site the whole time, sitting in his car in the parking lot, absent-mindedly engrossed in a telephone call.

  Magic Leap broke ground on the new 259,000-square-foot campus in October 2015, and when I visited a year later, Abovitz expected the majority of its 850 employees to move in before the end of 2016. The rest of its workers were scattered in nine offices around the globe, not just in tech hot spots like Silicon Valley and Austin but also in far-flung outposts like Wellington, New Zealand, and Tel Aviv. Some groups were already in place at the new site, including a machine shop and several engineering teams. It was important to Abovitz to keep critical development teams together as part of an “agile hardware” model, which allowed the company to produce “literally many hundreds of iterations” of its headset prototype. “Part of why Magic Leap can iterate so fast is because we have all the right people in the right places,” Abovitz said.

  The company was also building manufacturing facilities on the Plantation campus. “This is the most spaceshippy part of Magic Leap,” Abovitz said as he led me into the production line areas: a series of long self-contained modular bays lined up like submarines in a port. Each line could be activated as needed, ramping up production from thousands of units a year to more than a million.

  Abovitz wanted Magic Leap to stay in Florida. One of the benefits of manufacturing there was that it allowed the company to keep its secrets. If it had been headquartered in Northern California, that would be nearly impossible, given Silicon Valley’s job-hopping culture and well-lubricated rumor mill. Of course, Magic Leap would also have an easier time hiring, but so far the company’s technology had been a strong draw, pulling people in from the Valley and other tech hubs. “We’re bringing a crazy amount of very capable high-end engineering and manufacturing tech talent into Florida,” he said.

  So now that it had a production line, what was Magic Leap’s timing for entering the market? I asked. Abovitz was tight-lipped. “Soon-ish,” he said. He was also reticent about what the headset will cost. “Not a luxury product,” he finally offered.

  Magic Leap was developing much of its content in-house, and had already hired several well-known video game designers, cartoonists, artists, and writers. Neal Stephenson, author of Snow Crash, the seminal 1992 novel about virtual reality, was working as Magic Leap’s chief futurist and developing an undisclosed game. Other content was coming from partners like Weta Workshop, which operated a twenty-five-person lab with Magic Leap in New Zealand. Their first project, Dr. Grordbort’s Invaders, was an action-oriented game set in a steampunk version of a future British empire: the player used a ray gun to fight off evil robots that seem to break through the walls of their living room and fly around the house.

  Another high-profile partner allowed Magic Leap to work with characters that were close to Abovitz’s heart. In June 2016, the company announced a strategic partnership with ILMxLAB, the immersive entertainment division of Lucasfilm Ltd., and opened a joint research lab on Lucasfilm’s San Francisco campus. By the time I visited the Florida headquarters, the collaboration had already produced several mixed reality experiences set in the Star Wars universe, including the AT-AT attack that appeared to blow open a section of the building.

  “It’s like being in the earliest days of cinema, where people were sort of inching their way along in terms of understanding the format and what was possible and what was compelling,” said Vicki Dobbs Beck, the head of ILMxLAB.

  But entertainment was only the first part of Magic Leap’s plan—the foot in the door. Business applications would follow. Medical systems were likely to be a sweet spot, like 3-D radiology displays that float above patients in an operating room. Retail would be an early target too. “People love to go to a mall or store and see the things they’re going to buy in the flesh,” Abovitz said. “I think we can bring some aspects of that to your home that are impossible on your phone or tablet or TV.” Imagine a “try it on” button on your favorite online clothing store—click it and then look in a mirror, and the headset renders the outfit on your body so you can see how it looks.

  * * *

  —

  Naturally, Abovitz wasn’t the only entrepreneur who saw opportunity in this field—all the big names in tech were scrambling to get a piece of the action.

  At the time of my site visit, Magic Leap’s biggest competitor was Microsoft, which announced an augmented reality headset called HoloLens in 2014. A preproduction version of the hardware, the HoloLens Development Edition, had shipped to an unspecified number of Microsoft hardware and software developers in March 2016. Google had already dipped its toes into AR in 2013 with its ill-fated Google Glass, but its investment in Magic Leap indicated ongoing interest. Apple was working on AR as well, but it was unclear whether they were developing their own headset or adding capabilities to their iPhone line. And then there were a few other well-financed start-ups, most notably San Mateo, California’s Meta (which had raised $50 million in funding) and Mountain View, California–based Atheer (with $23 million raised), that were working on their own AR headsets.

  Magic Leap wouldn’t be the first or only company on the market. But its technology still seemed light-years ahead. Its success so far reminded me of another time a group of hardworking visionaries built something in Florida that rocked the world—and more than one Magic Leap employee compared what they were building in Plantation to NASA’s early days in Cape Canaveral.

  “To me, this feels like we’re crawling the rocket out to the pad,” Abovitz said.

  Epilogue

  THE AGE OF THE UNREAL

  During the years of reporting covered in the pages of this book, on perhaps a dozen occasions I experien
ced something I’ve come to describe as a “dinosaur in a hotel room” moment. It’s something that had never happened to me before, and something that’s hard to explain because it’s new and profound. But I think it’s something I’m starting to understand. I know other people have experienced it recently. And I suspect the rest of humanity is going to have to deal with it soon.

  Every time I’ve experienced one of these moments, I’ve had some kind of high-tech hardware strapped to my head. But the dinosaur in the hotel room is bigger than the thrill I get from seeing a convincing simulation, and stranger than getting an acute stress response just because a virtual Tyrannosaurus rex roared in my face.

  It’s the moment that comes just after that—a moment of conscious recognition that reality is not as it seems. When it happens, I know I’m in a simulation, I know what I’ve been experiencing isn’t actually real . . . but I simultaneously know that it is real, as real as anything I’ll experience after I take off the headset. I saw it, I felt it, I believed it, I’ll remember it just as vividly as any other memory. It happened. It doesn’t matter how.

  By the time this book is published, more than two years will have passed since a new generation—the first really good generation—of virtual reality headsets arrived in stores. From a business perspective, the Oculus Rift, the HTC Vive, the PlayStationVR, and all the others have been moderate successes. Smart people who track these things tell me that at the end of 2017, the VR and AR market was already worth more than $11 billion, and that it will be worth more than $200 billion in 2021. Not bad for an industry that barely existed five years earlier.