by Steve Krug
[ 15 ]
chapter 1
I have to admit I was a little anxious the first few times I did live demo tests for an audience. But I’ve probably done fifty of them by now and it’s worked every time, no matter what the site is and no matter who the participant is.
The fact is, it just works. Ask anyone who’s done any amount of usability testing and they’ll tell you that it pretty much always works. If you sit somebody—almost anybody—down and have them try to use what you’re building, they’ll inevitably encounter some of the problems that most people are going to encounter.
But why does it work?
It may not seem to make sense that something so simple (just giving people something to do and watching them do it) can consistently reveal serious usability problems. But if you think about it for a while (or for several years, in my case), there are reasons why it works:
It works because all sites have problems. We all know this from our own experience. How often have you used a Web site and not run into a usability problem? And they’re often significant problems that seriously frustrate you or even keep you from doing what you set out to do.
Some mature sites may have fewer serious problems, especially if they’ve been through repeated rounds of usability testing, but don’t kid yourself: Your site has usability problems. Heck, my site has usability problems, which as you can imagine is potentially quite embarrassing. Even Amazon has usability problems, and it’s common knowledge how highly I think of Amazon. 3
It works because most of the serious problems tend to be easy to find. Again, think about the usability problems you’ve run into on other people’s Web sites. Don’t you usually find yourself thinking “How can they possibly not know about this problem?” Many of the most serious problems are lying around in plain sight, and almost everybody will run into them.
3 People love to email me about problems they fi nd on Amazon.com, as though I could do something about them. I do have an Amazon Prime membership ($79 a year gives me
“free” second-day shipping), but that’s about the extent of my infl uence. And Amazon does so much usability testing that if there’s a problem, I’m sure it’s not because they’re not aware of it; they probably just haven’t decided what to do about it yet.
[ 16 ]
you don’t see any elephants around here, do you?
And yet on our own
sites, we somehow think
of them as being hard to
find. It always reminds
me of the Vietnam-era
Doonesbury cartoon
where Phred asks the
curator of a demolished
Cambodian museum if
it was destroyed during
DOONESBURY © 1973 G. B. Trudeau. Reprinted with
the secret bombings.
permission of UNIVERSAL PRESS SYNDICATE. All rights reserved.
The usability problems on your site may not be obvious to you, because you know how it works—or how it’s supposed to work. Most of your
users, on the other hand, don’t, and that makes all the difference.
Of course, there are also serious usability problems that are more “hidden,”
the kind that not as many people will run into. But unless you have substantial resources to devote to usability (for instance, it’s your full-time job), I strongly recommend focusing on getting rid of the obvious ones first. Most sites don’t even manage to accomplish that.
And finally:
It works because watching users makes you a better designer.
Even though terms like “user-centered design” and “user experience” are now in the vocabulary of most people working on Web sites, relatively few designers, developers, stakeholders, managers, and check-signers—
who all have a hand in the design process—have actually spent any time watching how people use Web sites. As a result, we end up designing for our abstract idea of users, based for the most part on ourselves.
Watching users makes you smarter about how people use things and
how things can be designed for use. I like to say that it informs your design intelligence, sort of the way travel is a broadening experience.
[ 17 ]
chapter 1
Why so little of it gets done
So, if it’s so easy and so valuable, why isn’t frequent usability testing a standard part of every Web project? Even today, very few organizations do any usability testing, and if they do, they usually only do it once, near the end of the project.
I think it’s largely because most people still haven’t had any firsthand experience with usability testing, so they don’t know how valuable it can be.
But even if they have, there’s no shortage of plausible reasons not to do it.
Lack of time, for instance. Testing seems like a lot of work, and most of us already have more on our plate than we can manage. Most Web development schedules are so tight that the prevailing attitude is “Let’s get it out the door, and we can tweak it later.”
And then there’s the natural—and nearly universal—reluctance to show our work before it’s finished. We always know that what we’re working on has problems, so why bother showing it to people and wasting our time (and theirs) having them tell us what we already know? (And who likes having the flaws in their work exposed in public, anyway?)
These are all quite reasonable, but as you’ll see, they’re not necessarily true.
FAQ
You’re talking about very small samples. Can’t we get more reliable information from things that gather data about a lot of people, like Web analytics?
Yes, Web analytics can give you a very accurate picture of what people are doing on your site (“72% of all visitors left the Home page after less than 5
seconds”). The sample size is very large (all of your users, in fact), the data is based on actual use, and the query tools allow you to pose almost any statistical question and get an answer immediately. And with the advent of Google Analytics at such an attractive price point (free), this kind of data is available to everyone.
[ 18 ]
you don’t see any elephants around here, do you?
The problem, though, as any usability professional will be happy to tell you, is that while analytics can tell you in great detail what people are doing on your site, they can’t tell you why they’re doing those things.
For instance, if people are spending a lot of time on a particular page, the statistics can’t tell you whether it’s because they found the content very useful and they’re busy reading it or because it makes no sense and they’re busy trying to figure it out.
Usability testing, on the other hand, excels at helping you understand why people are doing things.
When it comes to finding and fixing usability problems, if I had to choose between awesome analytics that could tell me exactly what my users are doing (but with no chance to know what they’re thinking while they’re doing it) or sitting with one user for an hour, with the ability to hear what he’s thinking and ask probing questions, I’d take the one user every time.
[ 19 ]
chapter
chapter1
2
I will now saw
my [lovely]
assistant in half
what a do-it-yourself test looks like
[ 20 ]
Is that all there is / my friend?
—REFRAIN OF THE ENNUI-DRENCHED
PEGGY LEE SONG “IS THAT ALL THERE IS?”
In the last chapter I described the demo tests I do in my workshops. Now you’re going to watch one of them.
In most ways it’s exactly like what you’re going to do with your own site (or application or whatever). The main difference is that in an actual test you’d be doing more tasks—or longer tasks—so the session would typically last for about an hour.
Go to www.rocketsurgerymadeeasy.com and watch the “Demo Test” file.
1. (If you can’t go online ri
ght now—for instance, if you’re reading this on an old-fashioned airplane that doesn’t have Wi-Fi access—don’t worry; just go on ahead to the next chapter for now. But do make a point of watching the demo test as soon as you have a chance.)
2. While you’re watching, keep in mind that at the end of the demo I’m going to ask you to make a list of the three usability problems you noticed that you think you would most want to fix if it was your site.
Is that all there is?
Yes, that’s about it. No magic, no special skills. Some participants will run into more problems and spark more insights, some less, but on average you can expect to learn a lot from each one.
FAQ
So, if you don’t mind my asking, why did you give this a whole chapter?
Because watching the demo test is important. I wanted to make sure you do it.
[ 21 ]
chapter
chapter3
3
A morning
a month,
that’s all we ask
a plan you can actually follow
[ 22 ]
a morning a month, that’s all we ask
A Can a Week, That’s All We Ask!
—HIGHLY SUCCESSFUL ADVERTISING SLOGAN
OF THE BLUE DIAMOND GROWERS
COOPERATIVE, CIRCA 1990
As I mentioned in Chapter 1, people have a lot of plausible reasons for not doing usability testing. But the main reason most people don’t do it is that they think it has to be a big production—what I refer to as the Big Honkin’ Test.
While teaching my workshops, I’ve worked out what I think is a reasonable plan that anyone—whether in a large organization or a one-person operation—
can afford to do and that will enable you to test what you’re building several times during the course of a project.
It’s doable, and it gets the job done. It uncovers as many problems as you can actually fix. And it keeps you focused on fixing the most important problems first.
I like to sum up my “master plan” this way:
A morning a month,
that’s all we ask.
Basically, it amounts to doing a round of testing
once a month, with three users.
On testing day, you do three tests in the morning
and then debrief over lunch. By the time lunch
is over, you’re done with usability testing for the
month, and you know what you’re going to fix
1
before the next round.
1 If you’re in an Agile development environment, don’t fret. See page 27.
[ 23 ]
chapter 3
There are two important words to focus on here:
Morning. Limiting the testing to half a day—which means testing with only three users—simplifies recruiting and means that more people can come and watch.
Month. “Monthly” turns out to be a good interval. It’s about as often as most teams can afford to do testing, and it identifies enough problems to keep you busy fixing things for the next month.
If you announce that on the third Thursday of each month you’re going to have people on-site for testing, you set up the expectation that people in your organization will show up to watch and that the development team will have something ready to test.
Making it a routine eliminates having to decide when to test; you just test whatever you’ll have ready on testing day. (If you have to think about when you’re going to test, you’re not going to end up testing as often.) Do-it-yourself vs. the Big Honkin’ Test
“A morning a month” isn’t just about scheduling; it’s also shorthand for keeping it as simple as possible so you do it often.
Do-it-yourself testing doesn’t do everything the Big Honkin’ Test does, but it produces the results you need at a price you can afford. Here’s an overview of the differences (all of the moving parts will be described in detail in later chapters):
[ 24 ]
a morning a month, that’s all we ask
THE BIG HONKIN’ TEST
DO-IT-YOURSELF TESTING
TIME SPENT FOR
1–2 days of tests, then a week to
One morning a month includes
EACH ROUND OF
prepare a briefing, followed by
testing, debriefing, and deciding
TESTING
some process to decide what
what to fix
to fix
By early afternoon, you’re done
with usability testing for the
month
WHEN DO YOU
When the site is nearly complete
Continually, throughout the
TEST?
development process
NUMBER OF
Typically only one or two per proj-
One every month
ROUNDS OF
ect, because of time and expense
TESTING
NUMBER OF
5–8, sometimes ten to convince a
Three
PARTICIPANTS
skeptical manager
IN EACH ROUND
WHO DO YOU
Recruit carefully to find people
Recruit loosely, if necessary
TEST WITH?
who are like your target audience
Doing testing frequently is more
important than testing “actual”
users
WHERE DO YOU
Held off-site, in a rented facility
Held on-site, with observers in
TEST?
with an observation room with a
any conference room using screen
one-way mirror
sharing software to watch
WHO WATCHES?
2–3 days of off-site testing means
Half day of on-site testing means
not many people will observe
more people can see the tests
firsthand
“live”
REPORTING
Someone takes at least a week to
A 1–2 page email summarizes deci-
prepare a briefing
sions made during the debriefing
WHO
The person running the tests
The entire development team
IDENTIFIES
usually analyzes the results and
and any interested stakeholders
THE
recommends changes
compare notes and decide what to
PROBLEMS?
fix over lunch the same day
[ 25 ]
chapter 3
THE BIG HONKIN’ TEST
DO-IT-YOURSELF TESTING
PRIMARY
A list of many problems (some-
A short list of the most serious
PURPOSE
times hundreds), categorized and
problems and a commitment to
prioritized by severity
fixing them before the next round
of testing
RECORD VIDEO
Yes. Observers need to see the
No. Seeing the participants’
OF THE PARTICI-
participants’ reactions to things
actions on the screen and hearing
PANT’S FACE?
(especially frustration)
them clearly is all that’s needed
OUT-OF-POCKET
$5,000 to $15,000 per round if
A few hundred dollars per round
COSTS
you hire someone to do it
FAQ
Can I really do this in a morning a month?
Well, no, not really; not all of it. What I’m saying is that the testing and the debriefing can all be done in a morning. And for most people on the team, that’s all the time they’ll have to spend each month.
But as the person running things, you’ll have some prep work to do for each round of testing
: deciding what to test, choosing the tasks, writing scenarios, recruiting participants, and getting all the stakeholders to attend.
The first time you do it, expect to spend at least two or three full days making all the preparations. For subsequent rounds, though, you should be able to whittle this down to two days, or even one.
Can I do it more often than once a month?
Definitely. A morning a month is just the minimum. Whatever you’re building will benefit from as much testing as you can manage to do.
The important thing, though, is not to do it less than once a month. As soon as you stop doing it on the third Thursday of each month, you’re back to making a decision about when to do it, which means that the odds of it happening drop dramatically.
[ 26 ]
a morning a month, that’s all we ask
We’re Agile. A morning a month? Ha!
Ah, yes. Agile. 2 Given the short cycles in an Agile environment, if you wait a month the world will have passed you by. Perhaps it’s more like
“A morning a sprint, that’s all we ask.”
In many ways, do-it-yourself testing is an excellent fit with Agile, which is based on rapidly producing working portions of the product and getting them in front of users. The only problem is that in many cases these
“users” are the team members who are doing the development. (You’re going to fix that.)
Since you’re going to be testing more than once a month, you may want to keep each round even leaner (two users instead of three, for instance) and do some of the rounds using remote testing (Chapter 14), which can save a lot of time. But other than that, the process is pretty much the same.
The biggest challenge with usability testing in an Agile environment seems to be that you need to be constantly laying out track ahead of the fast-moving programmers who don’t have time for prototyping. (Everything they write is assumed to be working code.)
This probably means that you’ll be spending part of your time creating prototypes of what they’ll be building in the next sprint. So in a given round, you’re likely to be testing what the team built in the previous sprint AND a paper prototype of what’s going to be built in the next one.
Does it have to be a morning?
There’s nothing magic about doing it in the morning. For instance, for some types of participants, it may be difficult for them to attend during work hours, so you might do tests at 6 pm, 7 pm, and 8 pm (providing dinner for the observers to encourage attendance) and then do the debriefing the next day over breakfast or lunch.