by Jeff Gothelf
As one of the main forms of digital payment in the world, PayPal is ubiquitous on the Internet. But unlike many Internet companies, PayPal works with money. This means that all their work must run through a thorough legal, risk, and regulatory compliance process. Is it possible to build and run regular experiments in this context? Can it be done with a complicated organizational structure and multiple stakeholders? The Hermes project proved that it could.
Fixing Checkout
In 2012, PayPal’s checkout process was stuck in the late 1990s, and the company was feeling the pinch. Complaints from customers were growing. Fast-growing startups like Square and Stripe threatened to drive down the payment giant’s market share. PayPal’s new President, David Marcus, who came from the startup world, saw an opportunity. Marcus wanted to reinvent Checkout, PayPal’s core workflow. He appointed industry veteran Bill Scott to run the project that became known as the Hermes project. His mission: to modernize the primary way in which online customers bought products and services with PayPal.
At that point at PayPal, a project of this magnitude would require a massive team and a timeline counted in years, not months or weeks. Marcus gave Scott six weeks to get it done. To help move it along, Marcus also gave the team his Balsamiq-created sketches for what the experience should entail. For Scott, a PayPal newcomer, one thing was clear: he would need to find a new way for PayPal to work. Needless to say this was an intimidating proposition.
The Team
The first item on the agenda was staffing. Scott, working with the product and design organization, gathered a core cross-functional team of eight people. This team of product managers, designers, and engineers was astonishingly small by PayPal standards at that time. A typical PayPal project then would have involved dozens of individuals and multiple locations.
All eight team members were staffed out of the same office except one—the designer (who commuted regularly by plane to be with the team in San Jose, California). With only six weeks to get the work done, there was no time for emails, product requirements documents, conference calls, and the delays that come from working across multiple sites and multiple time zones.
To ensure that their ideas met with risk and compliance guidelines the team augmented their staff, when needed, with representatives from those practices. Working with risk and compliance people throughout the project was important: this practice alone saved the team weeks of effort. On a normal project, teams wouldn’t meet with compliance until the end of the project. This big-batch approach to compliance caused delays and typically forced a flurry of last-minute product changes. The Hermes team, facing a looming deadline, couldn’t risk that kind of late-stage delay.
Getting Started and Overcoming Obstacles
Setting up shop in the nicest conference room on campus, the team spent their first week throwing out their best ideas, sketching, and coming up with an initial set of hypotheses. They then decided on a rhythm for the rest of their activities. The team settled on a schedule of designing and building on Monday through Wednesday, testing with users on Thursdays, and reflecting and planning on Fridays.
The first couple of weeks felt productive. Engineers took advantage of existing APIs that moved real money, designers sketched on the whiteboards, and frontend developers connected the pieces by implementing the work directly from the whiteboard. There were regular check-ins on status and the team built momentum and shared understanding.
Then, about four weeks into the project the designers revolted. Sharing their work constantly on the whiteboard and receiving ongoing feedback and critique from their colleagues began to feel like design by committee. “When can we go and ideate on our own for a bit?” they wondered. The team realized that just throwing a cross-functional array of professionals together in one room, although a good start, was not enough to build productive collaboration. The team needed to rethink how it worked and not just what it was doing.
The designers wanted to avoid what the team came to call “pissing on the Picasso,” a situation in which engineers respond to pixel-perfect mockups by debating what they can and can’t implement. The team refined their process to ensure that designers got some time to refine their thinking. They learned when to create sketches collaboratively and when to create and present pixel-perfect mockups. They figured out how to get thinking time, and at the same time, avoid going too long without broader team feedback. Over time, the Hermes team found their rhythm and shipped the first iteration successfully.
The Results
The Hermes team redefined the process of building products at PayPal. They created a model that the company could use to move away from silos, geographically distributed staffing, and lengthy debates and approval processes. By including previously excluded colleagues in their process, they redefined the way teams work with regulatory and risk to continue to ensure that they didn’t expose the company in a negative manner. The tweaks the team made to their design process allowed high-quality work to ship with less pushback from engineers. Finally, the things the Hermes engineering team learned laid the foundation for redesigning their tech stack to allow the rest of the organization to begin working with similar speed from prototype to production.
PayPal’s new standard practice is based on the Hermes model: it builds full-stack teams that are self-sufficient, colocated (or at least in the same time zone), and focused on rapid delivery and learning cycles.
The product the team created with this Lean approach was also a tremendous business success for PayPal. It became the core of their new checkout experience, which drove measurably higher customer success rates for sellers and buyers.
Online to Offline: Lean UX at CarMax
For companies that offer a purely digital service, Lean UX practice can be fairly straightforward. They avoid many of the tangles created by real-world interactions and bricks-and-mortar commerce. But what happens when you’re creating a multichannel service? This is the case at CarMax, a retailer that uses their online channel to drive people to their physical stores. How do you know your online service is working? How do you optimize an experience that involves online shopping, onsite browsing, and the integration of a large in-store sales force? This is exactly the challenge that CarMax faces on a daily basis.
As the largest used car retailer in the United States and a Fortune 500 company, CarMax relies heavily on both its website to inform and educate its customers about its inventory as well as its retail stores (160 of them and growing every month) to complete the actual sale.
Seeking an Outcome
In car sales, financing the purchase of the car is a critical component of the process.
CarMax wanted to optimize the online channel to ensure that customers who arrive at the store are not only educated but qualified and ready to buy. Could they do it? Archie Miller, UX designer, and Beth Sutherland, product manager, told us how their team discovered the answer.
The product team hypothesized that if customers better understood the financing side of purchasing a car, those customers would have a more successful car-buying experience when they arrived at the CarMax store. This project framing, seeking outcomes rather than committing to outputs, is at the heart of the Lean UX approach.
Lean UX + Customer Experience + Service Design
The team began its research by creating a customer journey map. To validate their thinking, they asked existing customers to create their own journey maps. CarMax’s journey map had 80 steps. The customer-created journey map had eight. This was a big insight: the customers’ world view was far less complex than the team had hypothesized.
Proto Personas
The biggest “A-ha!” moment this exercise revealed, though, had to do with how customers began their car-buying process. The exercise revealed a clear set of categories of factors that drove consumers into the car-buying process. For example, one persona that emerged (they called her Tiffany) was thrown into car buying through an unexpected life event like her car breaking down. The team fel
t this was an underserved segment.
They created other personas, as well, and they further validated these personas with a series of conversations CarMax ran using Ethnio to intercept shoppers on their website and engage in dialog. (Overall, the team spoke with more than 100 car shoppers to confirm these new findings.)
In addition, they learned two more important consumer insights about this group of customers:
Most customers who fell into this category did not enjoy the car-buying process.
Many of these shoppers faced a confidence gap about whether they would be approved for financing to purchase a CarMax vehicle.
Many people knew their overall credit profile, but didn’t know what it would mean for their ability to be approved for a loan. In their words, “Do I even qualify for a car loan with CarMax?”
The team set out to help their customers feel comfortable during their car-buying experience and ensure that they made the best purchase for their needs and budget while at the same time helping the business achieve its goals. The team began by creating empathy maps during each contextual interview with a customer. This helped team members realize insights and recognize consumer behavior patterns.
Figure 9-1. Empathy maps that the entire team created during customer interviews
Testing a Hypothesis
One of the first things the team wanted to test was whether the shortest loan application form would increase completion rates. They mocked up an initial draft in Axure, which you can see in Figure 9-2.
Figure 9-2. The first form the team created—the shortest possible form—in Axure
Using Ethnio to get users to work through their prototype, the team was surprised to learn that a short form, albeit simple and quick to complete, didn’t give users a sense of confidence that they were actually approved for a loan. This led many of the participants to believe that when they got to the store they’d have to go through loan approval again. These customers actually wanted to provide more information so that it “felt more like a loan application” than the prototype led them to believe.
The Next Iteration
The team set out to design a longer form with the internal challenge to not make it “feel” like a long form. Having a good sense of “Tiffany” and her needs, the team decided that a budget calculator was a good place to begin. This would allow Tiffany to determine whether she could afford the monthly car payment. This was followed by asking for the remaining information necessary to complete a credit application. These two elements together would reveal whether customers qualified for a loan, making the rest of the form process relevant or, in some cases, not worth the extra effort. Again, the team went to Ethnio and Axure to create and test the form. The end result (see Figure 9-3) was a “chunked-up” long form—a higher number of fields broken down into categories—that didn’t feel as long. Figure 9-4 shows the actual finished form.
Figure 9-3. The “long and chunked-up” prototype form that the team created in Axure
Figure 9-4. The first “live” form that shipped to real customers
Testing Another Hypothesis
Sometimes, it makes sense to use your research process to test more than one question at a time. That’s what the CarMax team did here. As the team refined their form design, they began testing a second hypothesis: would customers even want to be prequalified for a car loan before arriving at the store?
To answer this question, the team used a method called the “404 test” or “button to nowhere.” In this case, the team placed call-to-action buttons to apply for a car loan in various places on their mobile website. Then, they watched traffic logs to see what the rate of response was for these calls to action.
Clicking the first implementation of the call to action displayed a “feature coming soon” message. It wasn’t a great experience for the user, but it gave the team a good sense of where to put these elements and how best to word them. They refined the test to drive away from this “404-style page” toward a lead-generation form. This worked to both the users’ benefit—making them feel like they were making progress in their application—and the team’s—informing them on the best entry points for the loan application process and providing real customer data to executives to validate their design decisions.
Integrating In-Store Sales Staff
Including sales consultants from the beginning of the project allowed the product team to ensure that the design of the information being delivered to the store sales staff met their needs and allowed them to have more meaningful conversations with customers either on the phone or when they arrived at the store. For sales consultants in the stores these new loan applications provided a way to understand customers’ needs more effectively.
The team observed sales consultants at two different stores to understand how they used this information. The team came to understand what information was useful at various points of the customer interaction, and this helped them design the way the information was presented to sales people. Initially, the team’s new loan application presented information on a “decisions page.” This assumed that the customer had already selected a car. In reality, neither the sales consultant nor the customer were at that phase yet. The feedback from the store staff allowed the team to iterate their information design to be more in sync with the actual sales process.
By iterating and testing different ways to present the information for their sales colleagues, the team found the right combination of information and design to shift these conversations from “What kind of car would you like?”—a question many buyers don’t really have an answer for—to, “Let’s discuss the kind of cars you qualify for”—which is a far more productive conversation for both parties.
Regular Cadence with the Team
This project was a textbook example of a balanced team sharing the Lean UX work. Research sessions were scheduled consistently on the same day (Thursday) every week and invitations extended to the entire product team. Between each session, the team would stay engaged by conducting sketching sessions to refine the prototype and the testing script.
In addition, the team proactively communicated to colleagues through a variety of channels. They posted all of their findings on wall space in the office where many of their colleagues could see it. The team took advantage of CarMax’s culture of storytelling by creating the concept of “open houses” conducted by various product teams, as demonstrated in Figure 9-5. These public space events showcased a team’s work and allowed executives as well as other teams to get a sense of how projects were progressing, what was working, and what the team was shifting based on its learning.
Figure 9-5. Miller (closest to the board on the left) and Sutherland (right) sharing their findings at an open-house event
Setting Client Expectations at a Digital Product Studio: Lean UX at ustwo
There are many challenges when it comes to selling product development consulting. By definition, you’re trying to create something from nothing, so ambiguity and uncertainty are high. In this context, getting aligned on project scope and working process is critical. Add a new, unfamiliar process like Lean UX and things become even more complicated. In this case study, we look at how ustwo, a consulting firm with studios in London, New York, Malmö, and Sydney, has addressed this challenge.
There are lots of challenges at the beginning of a consulting engagement. You need to build a shared understanding of the goals of the engagement with your client. You need to set the appropriate expectations about what the agency is committed to achieving. You must ensure that your client understands how you plan to work and how you expect to collaborate. Finally, if you’re working in a Lean UX approach, you need to help your client understand and commit to a more outcome-driven result (rather than the more typical approach in which you agree to build a specific predefined set of features). Design studio ustwo has created and refined an approach to tackling these challenges head on.
The Service Definition Workshop
us
two has created a short preengagement phase they call a Service Definition Workshop. They sell this to every client before they commit to taking on the entire project. This workshop involves the team that will work on the full project and key stakeholders from the client. They spend one to two days together to lay out the assumptions of the project, the risks that are involved, what skills they’ll need, and what kind of support will be required from the client.
Using a series of facilitated brainstorming exercises, convergent and divergent thinking explorations, and affinity mapping techniques, ustwo introduces the concepts of Lean UX to the client, sets expectations about how much client participation will be necessary in the full project, and begins the trust-building process.
The Service Definition Workshop sets the scope of the work and expectations for how the team will address the work. It allows the newly formed client/agency team to create the following:
A shared understanding of the vision and goal of the project
A prioritized list of business goals
A clear sense of who the first users of the system will be
A proposed customer journey for those users
An initial set of desired features that the team feels are important