Like his mother, Wayne had found a clever way around chronic pain. In the process, he helped, finally, usher in the touchscreen as the dominant portal to computers, and he wrote the first dictionary for the gesture-based language we all now speak.
Which brings us back to Jobs’s claim that Apple invented multitouch. Is there any way to support such a claim? “They certainly did not invent either capacitive-touch or multitouch,” Buxton says, but they “contributed to the state of the art. There’s no question of that.” And Apple undoubtedly brought both capacitive touchscreens and multitouch to the forefront of the industry.
Apple tapped a half a century’s worth of touch innovation, bought out one of its chief pioneers, and put its own formidable spin on its execution. Still, one question remains: Why did it take so long for touch to become the central mode of human-machine interaction when the groundwork had been laid decades earlier? “It always takes that long,” Buxton says. “In fact, multitouch went faster than the mouse.”
Buxton calls this phenomenon the Long Nose of Innovation, a theory that posits, essentially, that inventions have to marinate for a couple of decades while the various ecosystems and technologies necessary to make them appealing or useful develop. The mouse didn’t go mainstream until the arrival of Windows 95. Before that, most people used the keyboard to type on DOS, or, more likely, they used nothing at all.
“The iPhone made a quantum leap in terms of being the first really successful digital device that had, for all intents and purposes, an analog interface,” Buxton says. He gets poetic when describing how multitouch translates intuitive movements into action: “Up until that point, you poked, you prodded, you bumped, you did all this stuff, but nothing flowed, nothing was animated, nothing was alive, nothing flew. You didn’t caress, you didn’t stroke, you didn’t fondle. You just push. You poke, poke, poke, and it went blip, flip, flip. Things jumped; they didn’t flow.”
Apple made multitouch flow, but they didn’t create it. And here’s why that matters: Collectives, teams, multiple inventors, build on a shared history. That’s how a core, universally adopted technology emerges—in this case, by way of boundary-pushing musical experimenters; smart, innovative engineers with eyes for efficiency; idealistic, education-obsessed CEOs; and resourceful scientists intent on creating a way to transcend their own injuries.
“The thing that concerns me about the Steve Jobs and Edison complex—and there are a lot of people in between and those two are just two of the masters—what worries me is that young people who are being trained as innovators or designers are being sold the Edison myth, the genius designer, the great innovator, the Steve Jobs, the Bill Gates, or whatever,” Buxton says. “They’re never being taught the notion of the collective, the team, the history.”
Back at CERN, Bent Stumpe made an impressively detailed case that his inventions had paved the way for the iPhone. The touchscreen report was published in 1973, and a year later, a Danish firm began manufacturing touchscreens based on the schematic. An American magazine ran a feature about it, and hundreds of requests for information poured in from the biggest tech companies of the day. “I went to England, I went to Japan, I went all over and installed things related to the CERN development,” Stumpe says. It seems entirely plausible that Stumpe’s touchscreen innovations were absorbed into the touchscreen bloodstream without anyone giving him credit or recompense. Then again, as with most sapling technologies, it’s almost impossible to tell which was first, or concurrent, or foundational.
After the tour, Stumpe invites me back to his home. As we leave, we watch a young man slinking down the sidewalk, head bent over his phone. Stumpe laughs and shakes his head with a sigh as if to say, All this for that?
All this for that, maybe. One of the messy things about dedicating your life to innovation—real innovation, not necessarily the buzzword deployed by marketing departments—is that, more often than not, it’s hard to see how, or if, those innovations play out. It may feed into a web so thick any individual threads are inscrutable, and it may contribute to the richness of the ideas “in the air.” Johnson, Theremin, Norris, Moog, Stumpe, Buxton, Westerman—and the teams behind them—who’s to say how and if the iPhone’s interface would feel without any of their contributions? Of course, it takes another set of skills entirely to develop a technology into a product that’s universally desirable, and to market, manufacture, and distribute that product—all of which Apple happens to excel at.
But imagine watching the rise of the smartphone and the tablet, watching the world take up capacitive touchscreens, watching a billionaire CEO step out onto a stage and say his company invented them—thirty years after you were certain you proved the concept. Imagine watching that from the balcony of your third-floor one-bedroom apartment in the suburbs of Geneva that you rent with your pension and having proof that your DNA is in the device but finding that nobody seems to care. That kind of experience, I’m afraid, is the lot of the majority of inventors, innovators, and engineers whose collective work wound up in products like the iPhone.
We aren’t great at conceiving of technologies, products, even works of art as the intensely multifaceted, sometimes generationally collaborative, efforts that they tend to be. Our brains don’t tidily compute such ecosystemic narratives. We want eureka moments and justified millionaires, not touched pioneers and intangible endings.
ii: Prototyping
First draft of the one device
Jony Ive had finally decided that the time was right. Maybe Jobs was in an unusually good mood when he dropped by the ID studio for one of his frequent visits. Maybe Ive felt the demos that the engineers and UI wizards had cooked up—the pinch to zoom, the rotating maps—were as good as they were going to get on a wonky, stitched-together rig. Either way, one day in the summer of 2003, Ive led Jobs into the user testing facility adjacent to his design studio, where he unveiled the ENRI project and gave him a hands-on demonstration of the powers of multitouch.
“He was completely unimpressed,” Ive said. “He didn’t see that there was any value to the idea. And I felt really stupid because I had perceived it to be a very big thing. I said, ‘Well, for example, imagine the back of a digital camera. Why would it have a small screen and all of these buttons? Why couldn’t it be all display?’ That was the first application I could think of on the spot, which is a great example of how early on this was.”
“Still he was very, very dismissive,” Ive said.
In Jobs’s defense, it was a table-sized contraption with a projector pointed at a white piece of paper. The Apple CEO looked for products, not science projects.
According to Ive, Jobs spent the next few days thinking it over, and evidently changed his mind. Soon, in fact, he decided that he loved it. Later—as we saw last chapter—he would publicly announce that Apple invented it. And then would go on to tell the journalist Walt Mossberg that he’d come up with the idea of doing a multitouch tablet himself. “I’ll actually tell you kind of a secret,” Jobs said. “I actually started on the tablet first. I had this idea of being able to get rid of the keyboard, to type on a multitouch glass display.” Yet Jobs likely didn’t even know about the touch-tablet project until Jony had given him a demonstration. And he’d initially rejected it.
“When he saw the first prototype,” Strickon says, “I think the quote was either ‘This thing is only good for reading my email on the toilet’ or the other thing I heard was that he wanted a device that he could [use to] read email on the toilet. It came out both ways.” Regardless, that became the product spec: Steve wanted a piece of glass he could read his email on.
At one point, the ENRI group was standing around the ID studio, when Greg Christie blew in. He’d been meeting with Jobs on a regular basis about multitouch. “Now what’s the latest from Steve on this?” someone asked.
“Well,” he said, “First thing everyone needs to note is: Steve invented multitouch. So everybody go back and change your notebooks.” And then he grinned.
They rolled their eyes and laughed—that was pure Steve Jobs. Even now, Huppi’s amused when the Mossberg incident comes up. “Steve said, ‘Yeah, I went to my engineers and said “I want a thing that does this this and this”’—and that’s all total bullshit because he had never asked for that.” No one heard Steve talk about multitouch before he saw the ENRI team’s demo. “As far as I know, Jony showed him the demo of multitouch and then it was clicking in his mind.… Steve does this, you know: He comes back later and it’s his idea. And no one’s going to convince him otherwise.” Huppi laughs. He doesn’t seem bitter; it was a fact of life at Jobs-led Apple. “And that’s fine.”
Jobs’s approval raised the profile of the project, and, unsurprisingly, stirred up interest inside the company. “These meetings were a lot bigger now,” Huppi says. A project was greenlit to translate the fragments, ideas, and ambitions of the ENRI experiments into a product. The hardware effort to transform the rig into a working prototype—which at the time was a multitouch tablet—was given a code name: Q79. The project went on lockdown.
And there was still a long way to go.
The rig still relied on a plastic FingerWorks pad, for one. “The next question was, How the hell would you do it on a clear screen?” Huppi says. “And we had no idea how that was going to happen.”
That’s because the FingerWorks pad was absolutely loaded with chips. “For every little five-millimeter-by-five-millimeter patch, there was an electrode going to a chip—and so there were a lot of chips to cover a whole device of that size,” Huppi says. That was fine for an opaque black pad where you could hide them—but how could they ever do that on glass with a screen underneath? Huppi wondered.
So, Josh Strickon hit the books, reading up on the touch tech literature, digging through papers and published experiments and tinkering with alternatives. Research at Apple wasn’t always easy; Steve Jobs had shut down the company library, which used to provide engineers and designers with an archival resource, after his return. Still, Strickon was resolute: “There had to be a better way.”
Smarter Skin
Soon, he had some good news. Strickon thought he’d found a solution that might let them do multitouch on glass without having to deal with an avalanche of chips.
“I found Sony SmartSkin,” Strickon says. Sony was in the process of becoming one of Apple’s chief competitors—it was losing market share in portable music players to Apple’s iPods. Sony had been digging into capacitive sensing too. “This paper from Sony implied that you could do true multitouch with rows and columns,” Strickon says. It meant a lattice of electrodes laid out on the screen could do the sensing.
Josh Strickon considers this to be one of the most crucial moments in the course of the project. The paper, he says, presented a “much more elegant way” to do multitouch. It just hadn’t been done on a transparent surface yet. So, tracing the outline of Sony’s SmartSkin, he patched together a DIY multitouch screen. “I built that first pixel with a sheet of glass and some copper tape. That is what kicked the whole thing off,” he says, bothered not one iota that the method was borrowed from the competition. “I came from a research background where you look at what is in the field.” That approach, and the prospect of building new products based on another company’s research, spooked Apple’s legal team. “Once things started getting going with multitouch, the lawyers instructed us not to do those sorts of searches anymore,” Strickon says, annoyed at the memory. “I am not sure how you are expected to innovate without having an understanding about what was done before.”
Regardless, that pixel was tangible proof that you wouldn’t need to load up a device with chips to do multitouch. The input team now had to expand that lonely pixel into a full, tablet-size panel. So they went on a shopping spree.
“We scrambled—we would grab some parts from RadioShack or wherever we had to get them, and we cooked this thing up on glass,” Huppi says. “It was a piece of glass with a couple of copper electrodes taped in there, and I mean totally cooked up, breadboard-style.” Breadboards are what engineers use to prototype electronics; they started out as actual breadboards, of the wooden, yeast-handling variety, on which radio tinkerers soldered wires and evolved into a standard tool engineers use to experiment.
Touch had never been done quite like this before. The team built three of those breadboards—large, poker-table-size arrays with their guts splayed out on top—to prove the rig would be capable of registering real interactions. There’s one known prototype left, evidence of the earliest stages of the iPhone’s evolution, and it’s tucked away in an office at Apple. I was shown a rare picture of that first breadboard—it looked like raw chipboards often do, like a green sound-mixing board with an inlaid screen, surrounded by a rigid sea of circuits.
In order to understand exactly how a user’s hand was interacting with the touchscreen, Strickon programmed a tool that created a visualization of the palm and fingers hitting the sensors in real time. “It was kind of like those bed of nails that you put your hand into,” Huppi says, that “creates this three-D image of your hand on the other side.” They called it the Multitouch Visualizer. “That was literally like the first thing we put on the screen,” Strickon says. According to Huppi, it’s still used to monitor touch sensors at Apple today.
Strickon took the opportunity to mess around with the musical capabilities of the new device too, and wrote a program that transformed the touchpad into a working theremin. Moving his hand left or right modulated pitch, while moving it up or down controlled volume. The ancestor of the iPhone could be played like a Russian proto-synth before it could do much of anything else. “We had some fun doing goofy stuff like that,” Huppi says.
The prospect was tantalizing: Not only could multitouch power a new kind of tablet that users could directly manipulate—one that was fun, efficient, and intuitive—but it could work on a whole suite of trackpads and input mechanisms for typical computers too. A phone was still the furthest thing from their minds.
The hardware team was putting together a working touchscreen. Jobs was enthusiastic. Industrial Design was looking into form-factor ideas. And the tablet was going to need a chip to run the touch sensor software that Strickon had cooked up.
The team had never designed a custom chip before, but their boss Steve Hotelling had, and he pushed forward. “He just said, ‘Yeah, no problem, we’ll just get some bids out there we’ll get a chip made.… It’s going to take about a million bucks; in eight months we’ll have a chip,’” Huppi says.
They settled on the Southern California chipmaker Broadcom. In an unusual move, Apple invited the company’s reps to come in and see the “magic” themselves. “I don’t think it’s ever been done since,” Huppi says. Hotelling thought that if the outside contractors saw the demo in action, they’d be excited by it and do the work better and faster. So, a small Broadcom team was led into the testing lab. “It was just this amazing moment to see how excited these guys were,” Huppi says. “In fact, one of those guys that’s now at Apple, he came from that team, and he still remembers that day.”
But it would be months before the chip was done.
“Meanwhile, there was again a big driving force to build more ‘Looks like, feels like’ prototypes,” Huppi says. As the project grew, more and more people wanted their hands on it. The team knew that if they didn’t deliver, executives might lose interest. “We ended up building this tethered display, which looked like an iPad but plugged into your computer,” Strickon says. “That’s one of the things I pushed for—like, we’ve got to build stuff to put in their hands.”
The first round of proto-tablets they delivered weren’t exactly stellar, however.
“There were prototypes that were built that were basically like tablets, Mac tablets—things that could barely have more than an hour’s battery life.” Strickon says with a laugh. “It had no usefulness.” Which was a bit more generous than Bas Ording’s assessment. He, after all, had to use them to work on the UI.
“The thing would overheat in that enclosure, and the battery life was like two minutes,” he says with a laugh.
The team built fifty or so prototypes—thick, white, tablet-looking ancestors of the iPad. That way, software designers could just jack the touchpad into Mac software without sacrificing any performance or power, and the UI team could keep working on perfecting the interface.
Culture Clash
As the project drew on, some members of the team chafed under Apple’s rigid culture. Strickon, for one, wasn’t used to the corporate hierarchy and staid atmosphere—he was an ambitious, unorthodox researcher and experimenter, recall, and still fresh out of MIT. His boss, Steve Hotelling, chastised him for interrupting superiors (like Hotelling) at meetings; Strickon, meanwhile, shot back that Hotelling was a square “company man.” Worse was the old boys’ club that had a stranglehold on decision making, he says. “A lot of people had been there for a long, long time. People like [senior vice president of marketing] Phil Schiller, they’ve been around forever—that club was already kind of established.” He felt like his ideas were dismissed at meetings. “It was like, ‘I see how it is out here—not everybody’s supposed to have ideas.’”
Outside of the project, Strickon was lonely and despondent. “I was trying to meet people, but it really wasn’t working.… Even HR was trying to help me out,” he says, “introducing me to people.”
Ording’s good nature was tested from time to time too. He’d been sitting in weekly meetings with the CEO but often found Jobs’s mean streak too much to handle. “There was a period of time,” Ording says, “that for a couple months or so, half a year or whatever, I didn’t go to the Steve meetings.” Jobs would chew out his colleagues in a mean-spirited way that made Ording not want to participate. “I just didn’t want to go. I was like, ‘No, Steve’s an asshole.’ Too many times he would be nasty for no good reason,” he says. “No one understood, because most people would die to go to these meetings—like, ‘Oh, it’s Steve.’ But I was tired of it.”
The One Device Page 10