by Jeff Gothelf
How might we create more capable and compelling mobile offerings so that our high school and university students increase their usage of the LMS through their preferred mobile device?
This version of the problem statement sets a clear issue for the team to solve. It provides the source of the concern and a sense of the impact it might have on the company’s business and ultimate success. Finally, it provides clear guidelines for how the team should proceed without dictating a specific set of features for them to build, instead opting for an outcome, a change in customer behavior, for them to achieve.
New product: problem statement template
Problem statements for new products are also made up of three elements (see Figure 3-3):
The current state of the market
The opportunity the business wants to exploit (i.e., where current solutions are failing)
A strategic vision for a product or service to address the market gap
Figure 3-3. Problem statement template for new products and services
Let’s continue our example from the educational technology space and write a new product problem statement:
The current state of the educational technology market has focused primarily on selling large installations to school systems focused on making teachers’ and administrators’ lives simpler. These services were created in a desktop world and serve only the providers of education, not the students. These services fail to capture the way incoming students use technology today—a mobile-first or, in some cases, mobile-only consumption pattern. Our new Learning Management System product will address this gap by building mobile-friendly learning experiences tailored to the way primary and secondary school students use technology today as well as how they learn.
Regardless of which type of product you’re working on or who created them, problem statements are also filled with assumptions. The team’s job is to dissect the problem statement into its core assumptions. Here’s how to do that.
Running the Exercise: Business Assumptions Exercise
We like to use this exercise (created by our friend Giff Constable) to facilitate the assumptions discussion.
Step 1: Complete the assumptions exercise individually
Each member of your team should prepare answers for the questions that follow on their own. This includes your clients.
Step 2: Share your answers
Get together with your team (and client) to kick off the new initiative. Go around the table sharing everyone’s answers to the assumptions exercise, question by question.
Step 3: Collect, organize, prioritize
Collect these answers on sticky notes or a whiteboard and sort them into themes. As a team, attempt to prioritize which themes are most important for each question. Don’t worry if you get to the end of the exercise without clear agreement on all of the answers. The goal is to collect statements that reflect what you and your team think might be true. If you have strong disagreement on a point, capture the different perspectives.
Assumptions Worksheets
You might discover that some of these questions don’t apply to your project. That’s OK—you can adapt the questions to your situation as you see fit. If you’re early in the life of your product, you’ll probably spend more time on the business assumptions. If you’ve got a mature product, you’ll probably focus your energies on the user assumptions. The point is to cast a broad net and look for assumptions in all dimensions of your project.
When you’ve completed the exercise, you will have a list of assumptions. Your next step is to assemble these assumptions into hypotheses.
Hypotheses
With our assumptions in hand, we’re ready to move to the next step: writing hypotheses. To do that, we transform our assumptions into a format that is easier to test: the hypothesis statement.
Generally, hypothesis statements use this format:
We believe [this statement is true].
We will know we’re [right/wrong] when we see the following feedback from the market:
[qualitative feedback] and/or [quantitative feedback] and/or [key performance indicator change].
You can see that this format has two parts. A statement of what you believe to be true, and statement of the market feedback you’re looking for to confirm that you’re right.
Expressing your assumptions this way turns out to be a powerful technique. It takes much of the subjective and political conversation out of the decision-making process and instead orients the team toward evidence collected from the market. In other words, it orients the team toward their users and customers.
Data-Informed Design versus Data-Driven Design
There’s been a lot of backlash in the design world about data-driven design. The argument is that by reducing every design decision to factors that can be measured, it takes the delight and soul out of our products. We actually agree with this perspective, which is why we think it’s so important to include qualitative targets in your success criteria. Qualitative insight helps us to understand the emotional aspects of product design. It provides the “why” to give context to the quantitative “what” insights provided by analytics tools. It gives us a sense of what’s driving the behavior and provides guidance for design improvements that improve the experience. This makes our customers as well as our business more successful. By balancing qualitative and quantitative insights, we are using data to inform rather than dictate our design decisions.
Hypotheses: Tactical and Testable
At their highest levels, hypotheses can seem daunting. Where do you begin? What do you test first? In these situations, we’ve found it helpful to focus our hypothesis writing on features. Figure 3-4 presents the format we recommend for articulating tactical, testable hypothesis statements.
Figure 3-4. Feature hypothesis template
The first field is completed with the outcome you’ve determined is your measure of success. This is the business outcome you’d like to achieve. The second field describes exactly which of your target users you believe you should focus on first. The third field speaks to the end goal, benefit, or the emotional state those customers will get if we design and implement our feature well. The final field speaks to the way we believe we should improve our product or the new features we’d like to build.
It’s easy to jump to features first. Most teams do this. By structuring your conversations with hypotheses you force the team to think through the context first, and especially, the problem you’re trying to solve. This constrains the space for determining which features to work on. Each assumption you put in place increasingly confines the team’s thinking to a more accurate set of potential features. This focus makes any feature discussion more productive, more targeted, and ultimately more relevant to your customers.
Getting from Problem Statement to Hypothesis
Let’s take a look at an example of how this works by going back to the problem statement we looked at earlier:
Our Learning Management System (LMS) was intended to provide a central platform to facilitate communication, student body management, and assessment for parents, students, and teachers in most educational contexts. We have observed new students both in the United States and abroad entering schools with a mobile-first or mobile-only mindset. We have also observed a significant rise in funding for competing educational technology startups catering to this new behavior model. Our products are heavy and not mobile-friendly which increases our risk of losing incoming cohorts of new students through decreased usage and dependency on the LMS.
How might we create more capable and compelling mobile offerings so that our high school and university students increase their usage of the LMS through their preferred mobile device?
The team working on this problem knows that if they don’t meet the needs of an increasingly mobile user base, they will lose business. They need to figure out how to increase mobile use of the system by students. The team’s first step is to declare their assumpt
ions and then write a set of testable hypotheses for this problem.
Completing Your Hypothesis Statements
To create your hypothesis statements, you will need to begin assembling the building blocks. Here is what you are going to want to put together:
The business outcomes you are trying to achieve
The users you are trying to service
The user outcomes that motivate them
The features you believe might work in this situation
After you have all of this raw material, you can put them all together into a set of statements. Let’s take a closer look at each of these elements.
Running the exercise: business outcomes
Business outcomes are your definition of done. They are the result your business seeks, and the measuring stick for success. When you manage with outcomes, the question isn’t, “Did you ship it?” Instead, the question is, “How did this create a good result for us, for the customer, or for us both?” When you’re creating hypotheses to test, you want to try to be very specific regarding the outcomes you are trying to create. We discussed earlier how Lean UX teams focus less on output (the documents, sketches, products, and features that we create) and more on the outcomes that these outputs create. Can we make it easier for people to log in to our site? Can we encourage more people to sign up? Can we encourage greater collaboration among system users?
Together with your team, look at the problem you are trying to solve. You probably have a few high-level outcomes that you are trying to create (increasing sign-ups, increasing usage, etc.). Consider how you can break down these high-level outcomes into smaller parts. What behaviors will predict greater usage? More visitors to the site? More downloads of your app? Increasing number of items in the shopping cart? Sometimes, it’s helpful to run a team brainstorm to create a list of possible outcomes that you believe will predict the larger outcome you seek.
In the example in Figure 3-5 (from Giff Constable), an executive leadership team brainstormed and then voted on which Key Performance Indicators (KPIs) the company should pursue next. After consolidating down to the list shown in the photo, each executive was given four M&M’s. As long as they managed not to eat their votes, these executives were able to vote with candy for each metric they felt was most important. Ties were broken by the CEO.
Figure 3-5. Brainstorming a list of possible outcomes
Another way to get to your initial business outcomes is to use a framework called “Startup Metrics for Pirates.” Created by Dave McClure, an early employee of PayPal and the founder of 500 Startups, this framework is based on a customer lifecycle funnel (Figure 3-6). New customers come into the top of the funnel and move through increasing stages of engagement with a business. It also talks about pirates, which is fun.
Figure 3-6. Dave McClure’s Startup Metrics for Pirates
The Startup Metrics for Pirates framework is condensed into the acronym AARRR (Pirates!) and, when expanded, looks like this:
Acquisition
Can we get customers to our new feature or product?
Activation
After we get them there, can we get them to use it?
Retention
Can we get them to use it again? And again?
Referral
Can we get them to tell their friends, colleagues, bosses, or others about it?
Revenue
Can we get them to pay us for this feature?
Each step in this framework indicates a greater level of engagement from your customers. And, in a digital product or service, these are all measurable behaviors, which means they serve as great business outcomes to determine the initial traction of your efforts and where to focus your optimization efforts.
Output, Outcome, and Impact
We’ve talked a lot in this chapter about outcomes as the measure of success for your product. One challenge we’ve observed with the teams we’ve worked with is finding the right level of granularity in determining the right outcomes to measure. To help with that, let’s look at three different measures of success:
Output
These are features we design, implement, and ship. As a measure of success they are very common because they are clearly visible and easy to measure (you either shipped the feature or you didn’t). What they don’t measure is value to the customer. They only capture the team’s delivery performance.
Outcome
This is the change in the world we hope to see after we’ve created the output. As a measure of success, these are rare primarily because they are not binary and instead operate on a sliding scale. If a team is asked to improve retention by 50% but only manages to improve it by 42%, does that mean they’ve failed? It’s not always clear.
Impact
These are high-level measures of business health. Most companies measure these in the form of revenue, profits, sales, Net Promoter Score, and so on. As a measure of team-level success, these are usually too high level because it is often difficult to attribute a direct correlation between the launch of a tactical feature or a system optimization and an impact-level improvement. There are far too many factors that regularly affect these measures.
It is therefore in your best interest to ask your teams to work on outcome-level metrics. It’s at this level of granularity that teams can draw direct correlation between work they are doing and explicit changes in customer behavior.
Running the exercise: users
If your team already has a well-defined set of personas, the only thing you need to consider at this point is which ones you will be using in your hypothesis statements. If it’s been a while since you last reviewed your personas, things might have changed. This makes for a great opportunity to ensure that they are still relevant and that you and your colleagues still believe they are representative of your target audience. If you don’t have personas yet, though, this section will tell you how we like to create personas for the Lean UX process.
Proto-Personas
Designers have long been advocates for the end user. Lean UX doesn’t change that. As we make assumptions about our business and the outcomes we’d like to achieve, we still need to keep the user front and center in our thinking.
Most of us learned to think about personas as a tool to represent what we learned in our research. And it was often the case that we created personas as the output of lengthy, expensive research studies. There are a few problems with personas that are created this way. First, we tend to regard them as untouchable because of all of the work that went into creating them. In addition, it’s often the case that these personas were created by a research team or third-party vendor. This creates a risky knowledge gap between the people who conducted the research and those who are using the personas.
In Lean UX, we change the order of operations in the persona process. We also change persona-creation from a one-time activity to an ongoing process—one that takes place whenever we learn something new about our users.
When creating personas in this approach, we start with assumptions and then do research to validate our assumption. Instead of spending months in the field interviewing people, we spend a few hours creating proto-personas. Proto-personas are our best guess as to who is using (or will use) our product and why. We sketch them on paper (Figure 3-7) with the entire team contributing—we want to capture everyone’s assumptions. Then, as we conduct ongoing research, we quickly find out how accurate our initial guesses are and we adjust our personas in response.
Figure 3-7. A proto-persona sketch
Besides putting the customer front and center for a diverse product development team, proto-personas serve two more key purposes:
Shared understanding
Imagine your team sitting around a table and someone says the word “dog.” What image comes to your mind? Is it the same image that comes to your colleagues’ minds (Figure 3-8)? How do you know?
The same thing happens when someone says “the customer.” The proto-pers
ona approach ensures that everyone has the same image in their head when “the user” is invoked.
Figure 3-8. Dogs. (We are indebted to our esteemed colleague Adrian Howard
for this concept.)
Remembering we are not the user
It is often easy to assume our users are like us—especially if we consume the products we make. The reality is that we have a level of understanding and tolerance for the digital ecosystem that our customers rarely share. Going through a proto-persona exercise puts the focus on external users, pushing the team further away from their personal preferences for the product.
Using Proto-Personas
A team we were working with in New York was building an app that improved the Community Supported Agriculture (CSA) experience for New York City residents. CSA is a program that allows city residents to pool their money and purchase an entire season’s worth of produce from a local farmer. The farmer then delivers his crops, weekly, to the members of the CSA. Many subscribers to the CSA are men and women in their late 20s and early 30s who need to juggle a busy work life, an active social life, and a desire to participate in the CSA.