The value of qualitative research
Qualitative research helps us understand the domain, context, and constraints of a product in different, more useful ways than quantitative research does. It also helps us identify patterns of behavior among users and potential users of a product much more quickly and easily than would be possible with quantitative approaches. In particular, qualitative research helps us understand:
Behaviors, attitudes, and aptitudes of potential product users
Technical, business, and environmental contexts — the domain — of the product to be designed
08_084113 ch04.qxp 4/3/07 6:02 PM Page 51
Chapter 4: Understanding Users: Qualitative Research
51
Vocabulary and other social aspects of the domain in question
How existing products are used
Qualitative research can also help the progress of design projects by:
Providing credibility and authority to the design team, because design decisions can be traced to research results
Uniting the team with a common understanding of domain issues and user concerns
Empowering management to make more informed decisions about product design issues that would otherwise be based on guesswork or personal preference It’s our experience that in comparison, qualitative methods tend to be faster, less expensive, and more likely to provide useful answers to important questions that lead to superior design:
How does the product fit into the broader context of people’s lives?
What goals motivate people to use the product, and what basic tasks help people accomplish these goals?
What experiences do people find compelling? How do these relate to the product being designed?
What problems do people encounter with their current ways of doing things?
The value of qualitative studies is not limited to helping support the design process.
In our experience, spending the time to understand the user population as human beings can provide valuable business insights that are not revealed through traditional market research.
In one particularly illustrative example, we were asked by a client to perform a user study for an entry-level consumer video-editing product for Windows users. An established developer of video-editing and -authoring software, the client had used traditional market research techniques to identify a significant business opportunity in developing a product for people who owned a digital video camera and a computer but hadn’t connected the two yet.
In the field, we conducted interviews with a dozen users in the target market. Our first discovery was not surprising — that the people who did the most taping and had the strongest desire to share edited versions of their videos were parents. The second discovery, however, was quite startling. Of the 12 people whose homes we visited, only one person had successfully connected his video camera to his computer, and he had relied on the IT guy at work to set it up for him. One of the
08_084113 ch04.qxp 4/3/07 6:02 PM Page 52
52
Part I: Understanding Goal-Directed Design
necessary preconditions of the success of the product was that people could actually get video onto their computers to edit, but at the time it was extremely difficult to get a FireWire or video capture card functioning properly on an Intel-based PC.
As a result of four days of research, we were able to help our client make a decision to put a hold on the product, which likely ended up saving them a considerable investment.
Types of qualitative research
Social science and usability texts are full of methods and techniques for conducting qualitative research, and readers are encouraged to explore this literature. In this chapter, we will focus specifically on techniques that have been proven effective in our practice over the last decade, occasionally drawing attention to similar techniques practiced in the design and usability fields at large. We will also try to avoid getting bogged down in theory, and instead will present these techniques from a pragmatic perspective. The qualitative research activities we have found to be most useful in our practice are:
Stakeholder interviews
Subject matter expert (SME) interviews
User and customer interviews
User observation/ethnographic field studies
Literature review
Product/prototype and competitive audits
Stakeholder interviews
Research for any new product design should start by understanding the business and technical context surrounding the product. In almost all cases, the reason a product is being designed (or redesigned) is to achieve one or several specific business outcomes (most commonly, to make money). It is the designers’ obligation to develop solutions without ever losing sight of these business goals, and it is therefore critical that the design team begin its work by understanding the opportunities and constraints that are behind the design brief.
As Donald Schön so aptly puts it, “design is a conversation with materials”1 This means that for a designer to craft an appropriate solution, he must understand the capabilities and limitations of the “materials” that will be used to construct the product, whether they be lines of code or extruded plastic.
08_084113 ch04.qxp 4/3/07 6:02 PM Page 53
Chapter 4: Understanding Users: Qualitative Research
53
Generally speaking, a stakeholder is anyone with authority and/or responsibility for the product being designed. More specifically, stakeholders are key members of the organization commissioning the design work, and typically include executives, managers, and representative contributors from development, sales, product management, marketing, customer support, design, and usability. They may also include similar people from other organizations in business partnership with the commissioning organization.
Interviews with stakeholders should occur before any user research begins because these discussions often inform how user research is conducted. Also, it is usually most effective to interview each stakeholder in isolation, rather than in a larger, cross-departmental group. A one-on-one setting promotes candor on the part of the stakeholder, and ensures that individual views are not lost in a crowd. (One of the most interesting things that can be discovered in such interviews is the extent to which everyone in a product team shares — or doesn’t share — a common vision.) Interviews need not last longer than about an hour, though follow-up meetings may be called for if a particular stakeholder is identified as an exceptionally valuable source of information.
The type of information that is important to gather from stakeholders includes:
Preliminary product vision — As in the fable of the blind men and the elephant, you may find that each business department has a slightly different and slightly incomplete perspective on the product to be designed. Part of the design approach must therefore involve harmonizing these perspectives with those of users and customers.
Budget and schedule — Discussions on this topic often provide a reality check on the scope of the design effort and provide a decision point for management if user research indicates a greater (or lesser) scope is required.
Technical constraints and opportunities — Another important determinant of design scope is a firm understanding of what is technically feasible given budget, time, and technology constraints. It is also often the case that a product is being developed to capitalize on a new technology. Understanding the opportunities underlying this technology can help shape the product’s direction.
Business drivers — It is important for the design team to understand what the business is trying to accomplish. This again leads to a decision point, should user research indicate a conflict between business and user needs. The design must, as much as possible, create a win-win situation for users, customers, and providers of the product.
08_084113 ch04.qxp 4/3/07 6:02 PM Page 54
54
Part I: Understanding Goal-Directed Design
Stakeholders
’ perceptions of the user — Stakeholders who have relationships with users (such as customer support representatives) may have important insights on users that will help you to formulate your user research plan. You may also find that there are significant disconnects between some stakeholders’ perceptions of their users and what you discover in your research. This information can become an important discussion point with management later in the process.
Understanding these issues and their impact on design solutions helps you as a designer to better develop a successful product. Regardless of how desirable your designs are to customers and users, without considering the viability and feasibility of the proposed solution there is no chance that the product will thrive.
Discussing these topics is also important to developing a common language and understanding among the design team, management, and engineering teams. As a designer, your job is to develop a vision that the entire team believes in. Without taking the time to understand everyone’s perspective, it is unlikely that they will feel that proposed solutions reflect their priorities. Because these people have the responsibility and authority to deliver the product to the real world, they are guaranteed to have important knowledge and opinions. If you don’t ask for it upfront, it is likely to be forced upon you later, often in the form of a critique of your proposed solutions.
Subject matter expert (SME) interviews
Early in a design project, it is often invaluable to identify and meet with several subject matter experts (SMEs) — experts on the domain within which the product will operate. Many SMEs were users of the product or its predecessors at one time and may now be trainers, managers, or consultants. Often they are experts hired by stakeholders, rather than stakeholders themselves. Similar to stakeholders, SMEs can provide valuable perspectives on a product and its users, but designers should be careful to recognize that SMEs represent a somewhat skewed perspective. Some points to consider about using SMEs are:
SMEs are often expert users. Their long experience with a product or its domain means that they may have grown accustomed to current interactions. They may also lean towards expert controls rather than interactions designed for perpetual intermediates. SMEs are often not current users of the product and may have more of a management perspective.
SMEs are knowledgeable, but they aren’t designers. They may have many ideas on how to improve a product. Some of these may be valid and valuable, but the most useful pieces of information to glean from these suggestions are the causative problems that lead to their proposed solutions. As with users, when you encounter a proposed solution, ask “how would that help you or the user?”
08_084113 ch04.qxp 4/3/07 6:02 PM Page 55
Chapter 4: Understanding Users: Qualitative Research
55
SMEs are necessary in complex or specialized domains. If you are designing for a technical domain such as medical, scientific, or financial services, you will likely need some guidance from SMEs, unless you are one yourself. Use SMEs to get information on industry best practices and complex regulations. SME knowledge of user roles and characteristics is critical for planning user research in complex domains.
You will want access to SMEs throughout the design process. If your product domain requires use of SMEs, you should be able to bring them in at different stages of the design to help perform reality checks on design details. Make sure that you secure this access in your early interviews.
Customer interviews
It is easy to confuse users with customers. For consumer products, customers are often the same as users, but in corporate or technical domains, users and customers rarely describe the same sets of people. Although both groups should be interviewed, each has its own perspective on the product that needs to be factored quite differently into an eventual design.
Customers of a product are those people who make the decision to purchase it. For consumer products, customers are frequently users of the product; although for products aimed at children or teens, the customers are parents or other adult supervisors of children. In the case of most enterprise, medical, or technical products, the customer is someone very different from the user — often an executive or IT manager — with distinct goals and needs. It’s important to understand customers and satisfy their goals in order to make a product viable. It is also important to realize that customers seldom actually use the product themselves, and when they do, they use it quite differently from the way their users do.
When interviewing customers, you will want to understand:
Their goals in purchasing the product
Their frustrations with current solutions
Their decision process for purchasing a product of the type you’re designing
Their role in installation, maintenance, and management of the product
Domain-related issues and vocabulary
Like SMEs, customers may have many opinions about how to improve the design of the product. It is important to analyze these suggestions, as in the case of SMEs, to determine what issues or problems underlie the ideas offered, because better, more integrated solutions may become evident later in the design process.
08_084113 ch04.qxp 4/3/07 6:02 PM Page 56
56
Part I: Understanding Goal-Directed Design
User Interviews
Users of a product should be the main focus of the design effort. They are the people who are personally utilizing the product to accomplish a goal (not their managers or support team). If you are redesigning or refining an existing product, it is important to speak to both current and potential users, that is, people who do not currently use the product but who are good candidates for using it in the future because they have needs that can be met with the product and are in the target market for the product. Interviewing both current and potential users illuminates the effect that experience with the current version of a product may have on how the user behaves and thinks about things.
Information we are interested in learning from users includes:
The context of how the product (or analogous system, if no current product exists) fits into their lives or workflow: when, why, and how the product is or will be used
Domain knowledge from a user perspective: What do users need to know to do their jobs?
Current tasks and activities: both those the current product is required to accomplish and those it doesn’t support
Goals and motivations for using their product
Mental model: how users think about their jobs and activities, as well as what expectations users have about the product
Problems and frustrations with current products (or an analogous system if no current product exists)
User observation
Most people are incapable of accurately assessing their own behaviors,2 especially when they are removed from the context of their activities. It is also true that out of fear of seeming dumb, incompetent, or impolite, many people may avoid talking about software behaviors that they find problematic or incomprehensible.
It then follows that interviews performed outside the context of the situations the designer hopes to understand will yield less-complete and less-accurate data. You can talk to users about how they think they behave, or you can observe their behavior first-hand. The latter route provides superior results.
08_084113 ch04.qxp 4/3/07 6:02 PM Page 57
Chapter 4: Understanding Users: Qualitative Research
57
Perhaps the most effective technique for gathering qualitative user data combines interviewing and observation, allowing the designers to ask clarifying questions and direct inquiries about situations and behaviors they observe in real time.
Many usability professionals make use of technological aides such as audio or video recorders to capture what users say and do. Interviewers must take care not to make these technologies too obtrusive; otherwise, the users will be distracted and behave differently than they would off-tape. In our practice, we’ve found that
a notebook and a camera allow us to capture everything we need without compromising the honest exchange of information. Typically, we won’t bring out the camera until we feel that we’ve established a good rapport with the interview subject, and then we use it to capture things about the environment that are difficult to jot in our notes. However, video, when used with care, can sometimes provide a powerful rhetorical tool for achieving stakeholder buy-in to contentious or surprising research results. Video may also prove useful in situations where note taking is difficult, such as in a moving car.
Literature review
In parallel with stakeholder interviews, the design team should review any literature pertaining to the product or its domain. This can and should include product marketing plans, brand strategy, market research, user surveys, technology specifications and white papers, business and technical journal articles, competitive studies, Web searches for related and competing products and news, usability study results and metrics, and customer support data such as call center statistics.
The design team should collect this literature, use it as a basis for developing questions to ask stakeholders and SMEs, and later use it to supply additional domain knowledge and vocabulary, and to check against compiled user data.
Alan Cooper, Robert Reinmann, David Cronin - About Face 3- The Essentials of Interaction Design (pdf) Page 11