by Steve Krug
Besides wasting time, these arguments create tension and erode respect among team members and can often prevent the team from making critical decisions.
Unfortunately, there are several forces at work in most Web teams that make these debates almost inevitable. In this chapter, I’ll describe these forces and explain what I think is the best antidote.
“Everybody likes ________.”
All of us who work on Web sites have one thing in common—we’re also Web users. And like all Web users, we tend to have strong feelings about what we like and don’t like about Web sites.
As individuals, we love pages with main menus across the top and submenus down the left side because they’re familiar and easy to use, or we hate them because they’re so boring. We love pages with large evocative images because they’re engaging, or we hate them because we just want to get to the content. We really enjoy using sites with ______, or we find ______ to be a royal pain.
And when we’re working on a Web team, it turns out to be very hard to check those feelings at the door.
The result is usually a room full of individuals with strong personal convictions about what makes for a good Web site.
And given the strength of these convictions—and human nature—there’s a natural tendency to project these likes and dislikes onto users in general: to think that most users like the same things we like. We tend to think that most users are like us.
It’s not that we think that everyone is like us. We know there are some people out there who hate the things we love—after all, there are even some of them on our own Web team. But not sensible people. And there aren’t many of them.
Farmers vs. cowmen
On top of this layer of personal passion, there’s another layer: professional passion. Like the farmers and the cowmen in Oklahoma!, the players on a Web team have very different perspectives on what constitutes good Web design based on what they do for a living.1
1 In the play, the thrifty, God-fearing, family-oriented farmers are always at odds with the freewheeling, loose-living cowmen. Farmers love fences, cowmen love the open range.
The ideal Web page as seen by someone whose job is...
It’s always seemed to me that these people probably have the jobs they do because of who they are. Designers, for instance, probably became designers because they enjoy pleasant visual experiences. They get visceral pleasure from looking at pages full of elegant type and subtle visual cues. There are endorphins involved.
And developers tend to like complexity. They enjoy figuring out how things work, reverse engineering them in their head, and looking for ideas they can use. Again, there are endorphins at work.
And because these reactions are happening at a brain-chemical level, it’s very difficult for them to imagine that everybody doesn’t feel exactly the same way.
The result is that designers want to build sites that look great, and developers want to build sites with interesting, original, ingenious features. I’m not sure who’s the farmer and who’s the cowman in this picture, but I do know that their differences in perspective often lead to conflict—and hard feelings—when it comes time to establish design priorities.
At the same time, designers and developers often find themselves siding together in another, larger clash between what Art Kleiner describes as the cultures of hype and craft.2
2 See “Corporate Culture in Internet Time” in strategy+business magazine at strategy-business.com/press/article/10374.
While the hype culture (upper management, marketing, and business development) is focused on making whatever promises are necessary to attract venture capital, revenue-generating deals, and users to the site, the burden of delivering on those promises lands on the shoulders of the craft culture artisans like the designers and developers.
This modern high-tech version of the perennial struggle between art and commerce (or perhaps farmers and cowmen vs. the railroad barons) adds another level of complexity to any discussions of usability issues—often in the form of apparently arbitrary edicts handed down from the hype side of the fence.3
3 I once saw a particularly puzzling feature on the Home page of a prominent—and otherwise sensibly designed—site. When I asked about it, I was told, “Oh, that. It came to our CEO in a dream, so we had to add it.” True story.
The myth of the Average User
The belief that most Web users are like us is enough to produce gridlock in the average Web design meeting. But behind that belief lies another one, even more insidious: the belief that most Web users are like anything.
As soon as the clash of personal and professional opinions results in a stalemate, the conversation usually turns to finding some way (whether it’s the opinion of an outside expert, published research, a survey, or focus groups) to determine what most users like or don’t like—to figure out what the Average Web User is really like. The only problem is, there is no Average User.
In fact, all of the time I’ve spent watching people use the Web has led me to the opposite conclusion:
ALL WEB USERS ARE UNIQUE AND ALL WEB USE IS BASICALLY IDIOSYNCRATIC
The more you watch users carefully and listen to them articulate their intentions, motivations, and thought processes, the more you realize that their individual reactions to Web pages are based on so many variables that attempts to describe users in terms of one-dimensional likes and dislikes are futile—and counter-productive.
And the worst thing about the myth of the Average User is that it reinforces the idea that good Web design is largely a matter of figuring out what people like. It’s an attractive notion: Either pull-downs are good (because most people like them), or they’re bad (because most people don’t). Stories should be on a single long page or they should be broken up into many shorter pages. Home page carousels, mega menus, rollovers, etc. are either good or bad, black or white.
The problem is there are no simple “right” answers for most Web design questions (at least not for the important ones). What works is good, integrated design that fills a need—carefully thought out, well executed, and tested.
That’s not to say that there aren’t some things you should never do, and some things you should rarely do. There are some ways to design Web pages that are clearly wrong. It’s just that they aren’t the things that Web teams usually argue about.
The antidote for religious debates
The point is, it’s not productive to ask questions like “Do most people like pull-down menus?” The right kind of question to ask is “Does this pull-down, with these items and this wording in this context on this page create a good experience for most people who are likely to use this site?”
And there’s really only one way to answer that kind of question: testing. You have to use the collective skill, experience, creativity, and common sense of the team to build some version of the thing (even a crude version), then watch some people carefully as they try to figure out what it is and how to use it.
There’s no substitute for it.
Where debates about what people like waste time and drain the team’s energy, usability testing tends to defuse most arguments and break impasses by moving the discussion away from the realm of what’s right or wrong and what people like or dislike and into the realm of what works or doesn’t work. And by opening our eyes to just how varied users’ motivations, perceptions, and responses are, testing makes it hard to keep thinking that all users are like us.
Can you tell that I think usability testing is a good thing?
The next chapter explains how to test your own site.
Chapter 9. Usability testing on 10 cents a day
KEEPING TESTING SIMPLE—SO YOU DO ENOUGH OF IT
Why didn’t we do this sooner?
—WHAT EVERYONE SAYS AT SOME POINT DURING THE FIRST USABILITY TEST OF THEIR WEB SITE
I used to get a lot of phone calls like this:
As soon as I’d hear “launching in two weeks” (or even “two months”) and “usability
testing” in the same sentence, I’d start to get that old fireman-headed-into-the-burning-chemical-factory feeling, because I had a pretty good idea of what was going on.
If it was two weeks, then it was almost certainly a request for a disaster check. The launch was fast approaching and everyone was getting nervous, and someone had finally said, “Maybe we better do some usability testing.”
If it was two months, then odds were that what they wanted was to settle some ongoing internal debates—usually about something like aesthetics. Opinion around the office was split between two different designs; some people liked the sexy one, some liked the elegant one. Finally someone with enough clout to authorize the expense got tired of the arguing and said, “All right, let’s get some testing done to settle this.”
And while usability testing will sometimes settle these arguments, the main thing it usually ends up doing is revealing that the things they were arguing about weren’t all that important. People often test to decide which color drapes are best, only to learn that they forgot to put windows in the room. For instance, they might discover that it doesn’t make much difference whether you go with cascading menus or mega menus if nobody understands the value proposition of your site.
I don’t get nearly as many of these calls these days, which I take as a good sign that there’s more awareness of the need to make usability part of every project right from the beginning.
Sadly, though, this is still how a lot of usability testing gets done: too little, too late, and for all the wrong reasons.
Repeat after me: Focus groups are not usability tests.
Sometimes that initial phone call is even scarier:
When the last-minute request is for a focus group, it’s usually a sign that the request originated in Marketing. If the Marketing people feel that the site is headed in the wrong direction as the launch date approaches, they may feel that their only hope of averting potential disaster is to appeal to a higher authority: market research. And one of the types of research they know best is focus groups. I’ve often had to work very hard to make clients understand that what they need is usability testing, not focus groups—so often that I finally made a short animated video about just how hard it can be (someslightlyirregular.com/2011/08/you-say-potato).
Here’s the difference in a nutshell:
In a focus group, a small group of people (usually 5 to 10) sit around a table and talk about things, like their opinions about products, their past experiences with them, or their reactions to new concepts. Focus groups are good for quickly getting a sampling of users’ feelings and opinions about things.
Usability tests are about watching one person at a time try to use something (whether it’s a Web site, a prototype, or some sketches of a new design) to do typical tasks so you can detect and fix the things that confuse or frustrate them.
The main difference is that in usability tests, you watch people actually use things, instead of just listening to them talk about them.
Focus groups can be great for determining what your audience wants, needs, and likes—in the abstract. They’re good for testing whether the idea behind your site makes sense and your value proposition is attractive, to learn more about how people currently solve the problems your site will help them with, and to find out how they feel about you and your competitors.
But they’re not good for learning about whether your site works and how to improve it.
The kinds of things you learn from focus groups—like whether you’re building the right product—are things you should know before you begin designing or building anything, so focus groups are best used in the planning stages of a project. Usability tests, on the other hand, should be used through the entire process.
Several true things about usability testing
Here are the main things I know about usability tests:
If you want a great site, you’ve got to test. After you’ve worked on a site for even a few weeks, you can’t see it freshly anymore. You know too much. The only way to find out if it really works is to watch other people try to use it.
Testing reminds you that not everyone thinks the way you do, knows what you know, and uses the Web the way you do.
I used to say that the best way to think about testing is that it’s like travel: a broadening experience. It reminds you how different—and the same—people are and gives you a fresh perspective on things.1
1 As the Lean Startup folks would say, it gets you out of the building.
But I finally realized that testing is really more like having friends visiting from out of town. Inevitably, as you make the rounds of the local tourist sites with them, you see things about your hometown that you usually don’t notice because you’re so used to them. And at the same time, you realize that a lot of things that you take for granted aren’t obvious to everybody.
Testing one user is 100 percent better than testing none. Testing always works, and even the worst test with the wrong user will show you important things you can do to improve your site.
When I teach workshops, I make a point of always doing a live usability test at the beginning so that people can see that it’s very easy to do and it always produces valuable insights.
I ask for a volunteer to try to perform a task on a site belonging to one of the other attendees. These tests last less than fifteen minutes, but in that time the person whose site is being tested usually scribbles several pages of notes. And they always ask if they can have the recording of the test to show to their team back home. (One person told me that after his team saw the recording, they made one change to their site which they later calculated had resulted in $100,000 in savings.)
Testing one user early in the project is better than testing 50 near the end. Most people assume that testing needs to be a big deal. But if you make it into a big deal, you won’t do it early enough or often enough to get the most out of it. A simple test early—while you still have time to use what you learn from it—is almost always more valuable than an elaborate test later.
Part of the conventional wisdom about Web development is that it’s very easy to go in and make changes. The truth is, it’s often not that easy to make changes—especially major changes—to a site once it’s in use. Some percentage of users will resist almost any kind of change, and even apparently simple changes often turn out to have far-reaching effects. Any mistakes you can correct early in the process will save you trouble down the line.
Do-it-yourself usability testing
Usability testing has been around for a long time, and the basic idea is pretty simple: If you want to know whether something is easy enough to use, watch some people while they try to use it and note where they run into problems.
In the beginning, though, usability testing was a very expensive proposition. You had to have a usability lab with an observation room behind a one-way mirror and video cameras to record the users’ reactions and the screen. You had to pay a usability professional to plan and facilitate the tests for you. And you had to recruit a lot of participants2 so you could get results that were statistically significant. It was Science. It cost $20,000 to $50,000 a shot. It didn’t happen very often.
2 We call them participants rather than “test subjects” to make it clear that we’re not testing them; we’re testing the site.
Then in 1989 Jakob Nielsen wrote a paper titled “Usability Engineering at a Discount” and pointed out that it didn’t have to be that way. You didn’t need a usability lab, and you could achieve the same results with far fewer participants. The price tag dropped to $5,000 to $10,000 per round of testing.
The idea of discount usability testing was a huge step forward. The only problem is that every Web site (and app) needs testing and $5,000 to $10,000 is still a lot of money, so it doesn’t happen nearly often enough.
What I’m going to commend to you in this chapter is something even simpler (and a lot less expensive): Do-it-yourself usability testing.
I’m going to explain how
you can do your own testing when you have no time and no money.
Don’t get me wrong: If you can afford to hire a professional to do your testing, do it. Odds are they’ll be able to do a better job than you can. But if you can’t hire someone, do it yourself.
I believe in the value of this kind of testing so much that I wrote an entire (short) book about how to do it. It’s called Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems.
It covers the topics in this chapter in a lot more detail and gives you step-by-step directions for the whole process.
How often should you test?
I think every Web development team should spend one morning a month doing usability testing.
In a morning, you can test three users, then debrief over lunch. That’s it. When you leave the debriefing, the team will have decided what you’re going to fix before the next round of testing, and you’ll be done with testing for the month.3
3 If you’re doing Agile development, you’ll be doing testing more frequently, but the principles are still the same. For instance, you might be testing with two users every two weeks. Creating a fixed schedule and sticking to it is what’s important.
Why a morning a month?
It keeps it simple so you’ll keep doing it. A morning a month is about as much time as most teams can afford to spend doing testing. If it’s too complicated or time-consuming, it’s much more likely that you won’t make time for it when things get busy.