The One Device

by Brian Merchant

  The PLATO IV system would remain in use until 2006; the last system was shut down a month after Norris passed away.

  There’s an adage that technology is best when it gets out of the way, but multitouch is all about refining the way itself, improving how thoughts, impulses, and ideas are translated into computer commands. Through the 1980s and into the 1990s, touch technology continued to improve, primarily in academic, research, and industrial settings. Motorola made a touchscreen computer that didn’t take off; so did HP. Experimentation with human-machine interfaces had grown more widespread, and multitouch capabilities on experimental devices like Buxton’s tablet at the University of Toronto were becoming more fluid, accurate, and responsive.

  But it’d take an engineer with a personal stake in the technology—a man plagued by persistent hand injuries—to craft an approach to multitouch that would finally move it into the mainstream. Not to mention a stroke of luck or two to land it into the halls of one of the biggest technology companies in the world.

  In his 1999 PhD dissertation, “Hand Tracking, Finger Identification, and Chordic Manipulation on a Multi-Touch Surface,” Wayne Westerman, an electrical engineering graduate student at the University of Delaware, included a strikingly personal dedication.

  This manuscript is dedicated to:

  My mother, Bessie,

  who taught herself to fight chronic pain in numerous and clever ways,

  and taught me to do the same.

  Wayne’s mother suffered from chronic back pain and was forced to spend much of her day bedridden. But she was not easily discouraged. She would, for instance, gather potatoes, bring them to bed, peel them lying down, and then get back up to put them on to boil in order to prepare the family dinner. She’d hold meetings in her living room—she was the chair of the American Association of University Women—over which she would preside while lying on her back. She was diligent, and she found ways to work around her ailment. Her son would do the same. Without Wayne’s—and Bessie’s—tactical perseverance in the face of chronic pain, in fact, multitouch might never have made it to the iPhone.

  Westerman’s contribution to the iPhone has been obscured from view, due in no small part to Apple’s prohibitive nondisclosure policies. Apple would not permit Westerman to be interviewed on the record. However, I spoke with his sister, Ellen Hoerle, who shared the Westerman family history with me.

  Born in Kansas City, Missouri, in 1973, Wayne grew up in the small town of Wellington, which is about as close to the actual middle of America as you can possibly get. His sister was ten years older. Their parents, Bessie and Howard, were intellectuals, a rare breed in Wellington’s rural social scene. Howard, in fact, was pushed out of his first high-school teaching job for insisting on including evolution in the curriculum.

  Early on, Wayne showed an interest in tinkering. “They bought him just about every Lego set that had ever been created,” Hoerle says, and his parents started him on piano when he was five. Tinkering and piano, she says, are the two things that opened up his inventive spirit. They’d set up an electric train set in the living room, where it’d run in a loop, winding through furniture and around the room. “They thought, This kid’s a genius,” Hoerle says. And Wayne was indeed excelling. “I could tell when he was five years old that he could learn faster than some of my peers,” she recalls. “He just picked things up so much faster than everybody else. They had him reading the classics and subscribed to Scientific American.”

  Bessie had to have back surgery, which marked the beginning of a lifelong struggle. “That’s another thing that was very important about our family. A year after that, she basically became disabled with chronic pain,” Hoerle says. Ellen, now a teenager, took charge of “the physical side of motherhood” for Wayne. “I had to kind of raise him. I had to keep him out of trouble.”

  When Ellen went off to college, it left her brother isolated. He already didn’t relate particularly well to other kids, and now he had to do the household work his sister used to. “Cooking, cleaning, sorting laundry, all things he had to take over when he was eight.” By his early teens, Westerman was trying to invent things of his own, working with the circuits and spare parts at his father’s school. His dad bought kits designed to teach children about circuits and electricity, and Wayne would help repair the kits, which the high-school kids tore through.

  He graduated valedictorian and accepted a full-ride to Purdue. There, he was struck by tendinitis in his wrists, a repetitive strain injury that would afflict him for much of his life. His hands started to hurt while he was working on papers, sitting perched in front of his computer for hours. Instead of despairing, he tried to invent his way to a solution. He took the special ergonomic keyboards made by a company called Kinesis and attached rollers that enabled him to move his hands back and forth as he typed, reducing the repetitive strain. It worked well enough that he thought it should be patented; the Kansas City patent office thought otherwise. Undeterred, Wayne trekked to Kinesis’s offices in Washington, where the execs liked the concept but felt, alas, that it would be too expensive to manufacture.

  He finished Purdue early and followed one of his favorite professors, Neal Gallagher, to the University of Delaware. At the time, Wayne was interested in artificial intelligence, and he set out to pursue his PhD under an accomplished professor, Dr. John Elias. But as his studies progressed, he found it difficult to narrow his focus.

  Meanwhile, Westerman’s repetitive strain injuries had returned with a vengeance. Some days he physically couldn’t manage to type much more than a single page.

  “I couldn’t stand to press the buttons anymore,” he’d say later. (Westerman has only given a handful of interviews, most before he joined Apple, and the quotes that follow are drawn from them.) Out of necessity, he started looking for alternatives to the keyboard. “I noticed my hands had much more endurance with zero-force input like optical buttons and capacitive touch pads.”

  Wayne started thinking of ways he could harness his research to create a more comfortable work surface. “We began looking for one,” he said, “but there were no such tablets on the market. The touch pad manufacturers of the day told Dr. Elias that their products could not process multi-finger input.

  “We ended up building the whole thing from scratch,” Westerman said. They shifted the bulk of their efforts to building the new touch device, and he ended up “sidetracked” from the original dissertation topic, which had been focused on artificial intelligence. Inspiration had struck, and Wayne had some ideas for how a zero-force, multi-finger touchpad might work. “Since I played piano,” he said, “using all ten fingers seemed fun and natural and inspired me to create interactions that flowed more like playing a musical instrument.”

  Westerman and Elias built their own key-free, gesture-recognizing touchpad. They used some of the algorithms they developed for the AI project to recognize complex finger strokes and multiple strokes at once. If they could nail this, it would be a godsend for people with RSIs, like Wayne, and perhaps a better way to input data, period.

  But it struck some of their colleagues as a little odd. Who would want to tap away for an extended period on a flat pad? Especially since keyboards had already spent decades as the dominant human-to-computer input mechanism. “Our early experiments with surface typing for desktop computers were met with skepticism,” Westerman said, “but the algorithms we invented helped surface typing feel crisp, airy, and reasonably accurate despite the lack of tactile feedback.”

  Dr. Elias, his adviser, had the skill and background necessary to translate Wayne’s algorithmic whims into functioning hardware. Neal Gallagher, who’d become chair of the department, ensured that the school helped fund their early prototypes. And Westerman had received support from the National Science Foundation to boot.

  Building a device that enabled what would soon come to be known as multitouch took over Westerman’s research and became the topic of his dissertation. His “novel input integration techni
que” could recognize both single taps and multiple touches. You could switch seamlessly between typing on a keyboard and interacting with multiple fingers with whatever program you were using. Sound familiar? The keyboard’s there when you need it and out of the way when you don’t.

  But Wayne’s focus was on building an array of gestures that could replace the mouse and keyboard. Gestures like, say, pinching the pad with your finger and thumb to—okay, cut at the time, not zoom. Rotating your fingers to the right to execute an open command. Doing the same to the left to close. He built a glossary of those gestures, which he believed would help make the human-computer interface more fluid and efficient.

  Westerman’s chief motivator still was improving the hand-friendliness of keyboards; the pad was less repetitive and required lighter keystrokes. The ultimate proof was in the three-hundred-plus-page dissertation itself, which Wayne had multitouched to completion. “Based upon my daily use of a prototype to prepare this document,” he concluded, “I have found that the [multitouch surface] system as a whole is nearly as reliable, much more efficient, and much less fatiguing than the typical mouse-keyboard combination.” The paper was published in 1999. “In the past few years, the growth of the internet has accelerated the penetration of computers into our daily work and lifestyles,” Westerman wrote. That boom had turned the inefficiencies of the keyboard into “crippling illnesses,” he went on, arguing, as Apple’s ENRI team would, that “the conventional mechanical keyboard, for all of its strengths, is physically incompatible with the rich graphical manipulation demands of modern software.” Thus, “by replacing the keyboard with a multitouch-sensitive surface and recognizing hand motions… hand-computer interaction can be dramatically transfigured.” How right he was.

  The success of the dissertation had energized both teacher and student, and Elias and Westerman began to think they’d stumbled on the makings of a marketable product. They patented the device in 2001 and formed their company, FingerWorks, while still under the nurturing umbrella of the University of Delaware. The university itself became a shareholder in the start-up. This was years before incubators and accelerators became buzzwords—outside of Stanford and MIT, there weren’t a lot of universities providing that sort of support to academic inventors.

  In 2001, FingerWorks released the iGesture NumPad, which was about the size of a mousepad. You could drag your fingers over the pad, and sensors would track their movements; gesture recognition was built in. The pad earned the admiration of creative professionals, with whom it found a small user base. It made enough of a splash that the New York Times covered the release of FingerWorks’ second product: the $249 TouchStream Mini, a full-size keyboard replacement made up of two touchpads, one for each hand.

  “Dr. Westerman and his co-developer, John G. Elias,” the newspaper of record wrote, “are trying to market their technology to others whose injuries might prevent them from using a computer.” Thing was, they didn’t have a marketing department.

  Nonetheless, interest in the start-up slowly percolated. They were selling a growing number of pads through their website, and their dedicated users were more than just dedicated; they took to calling themselves Finger Fans and started an online message board by the same name. But at that point, FingerWorks had sold around fifteen hundred touchpads.

  At an investment fair in Philadelphia, they caught the attention of a local entrepreneur, Jeff White, who had just sold his biotech company. He approached the company’s booth. “So I said, ‘Show me what you have,’” White later said in an interview with Technical.ly Philly. “He put his hand on his laptop and right away, I got it… Right away I got the impact of what they were doing, how breakthrough it was.” They told him they were looking for investors.

  “With all due respect,” White told them, “you don’t have a management team. You don’t have any business training. If you can find a management team, I’ll help you raise the rest of the money.” According to White, the FingerWorks team essentially said, Well, you just sold your company—why not come run ours? He said, “Make me a cofounder and give me founder equity,” and he’d work the way they did—he wouldn’t take a salary. “It was the best decision I ever made,” he said.

  White hatched a straightforward strategy. Westerman had carpal tunnel syndrome, so his primary aim was to help people with hand disabilities. “Wayne had a very lofty and admirable goal,” White said. “I just want to see it on as many systems as possible and make some money on it. So I said, ‘If we sold the company in a year, you’d be OK with that?’” White set up meetings with the major tech giants of the day—IBM, Microsoft, NEC, and, of course, Apple. There was interest, but none pulled the trigger.

  Meanwhile, FingerWorks continued its gradual ascent; its customer base of Finger Fans expanded and the company began collecting mainstream accolades. At the beginning of 2005, FingerWorks’ iGesture pad won the Best of Innovation award at CES, the tech industry’s major annual trade show.

  Still, at the time, Apple execs weren’t convinced that FingerWorks was worth pursuing—until the ENRI group decided to embrace multitouch. Even then, an insider at Apple at the time who was familiar with the deal tells me that the executives gave FingerWorks a lowball offer, and the engineers initially said no. Steve Hotelling, the head of the input group, had to personally call them up and make his case, and eventually they came around.

  “Apple was very interested in it,” White said. “It turned from a licensing deal to an acquisition deal pretty quickly. The whole process took about eight months.”

  As part of the deal, Wayne and John would head west to join Apple full-time. Apple would obtain their multitouch patents. Jeff White, as co-founder, would enjoy a considerable windfall. But Wayne had some reservations about selling FingerWorks to Apple, his sister suggests. Wayne very much believed in his original mission—to offer the many computer users with carpal tunnel or other repetitive strain injuries an alternative to keyboards. He still felt that FingerWorks was helping to fill a void and that in a sense he’d be abandoning his small but passionate user base.

  Sure enough, when FingerWorks’ website went dark in 2005, a wave of alarm went through the Finger Fans community.

  One user, Barbara, sent a message to the founder himself and then posted to the group.

  Just received a (very prompt) reply for my email to Wayne Westerman, in which I asked him: “Have you sold the company and will your product line be taken up and continued by another business?” Westerman wrote back: “I wish manufacturing had continued or shutdown had gone smoother, but if we all cross our fingers, maybe the basic technology will not disappear forever. :-)”

  When the iPhone was announced in 2007, everything suddenly made sense. Apple filed a patent for a multitouch device with Westerman’s name on it, and the gesture-controlled multitouch technology was distinctly similar to FingerWorks’. A few days later, Westerman underlined that notion when he gave a Delaware newspaper his last public interview: “The one difference that’s actually quite significant is the iPhone is a display with the multi-touch, and the FingerWorks was just an opaque surface,” he said. “There’s definite similarities, but Apple’s definitely taken it another step by having it on a display.”

  The discontinued TouchStream keyboards became highly sought after, especially among users with repetitive strain injuries. On a forum called Geekhack, one user, Dstamatis, reported paying $1,525 for the once-$339 keyboard: “I’ve used Fingerworks for about 4 years, and have never looked back.” Passionate users felt that FingerWorks’ pads were the only serious ergonomic alternative to keyboards, and now that they’d been taken away, more than a few Finger Fans blamed Apple. “People with chronic RSI injuries were suddenly left out in the cold, in 2005, by an uncaring Steve Jobs,” Dstamatis wrote. “Apple took an important medical product off the market.”

  No major product has emerged to serve RSI-plagued computer users, and the iPhone and iPad offer only a fraction of the novel interactivity of the original pad
s. Apple took FingerWorks’ gesture library and simplified it into a language that a child could understand—recall that Apple’s Brian Huppi had called FingerWorks’ gesture database an “exotic language”—which made it immensely popular. Yet if FingerWorks had stayed the course, could it have taught us all a new, richer language of interaction? Thousands of FingerWorks customers’ lives were no doubt dramatically improved. In fact the ENRI crew at Apple might never have investigated multitouch in the first place if Tina Huang hadn’t been using a FingerWorks pad to relieve her wrist pain. Then again, the multitouch tech Wayne helped put into the iPhone now reaches billions of people, as it’s become the de facto language of Android, tablets, and trackpads the world over. (It’s also worth noting that the iPhone would come to host a number of accessibility features, including those that assist the hearing and visually impaired.)

  Wayne’s mother passed away in 2009, from cancer. His father passed a year later. Neither owned an iPhone—his father refused to use cell phones as a matter of principle—though they were proud of their son’s achievements. In fact, so is all of Wellington. Ellen Hoerle says the small town regards Wayne as a local hero.