by Steve Krug
you test and your audience. I favor this approach for three reasons: professionals from a specific domain, like cardiologists for instance. I like to So what would
Also, since you will have fixed the problems you uncovered in the first round, in I think I’d
you do next?
darned!
these files are very valuable because they’re much easier to review quickly than some others.
offer people a little more than the going rate, since (a) it makes it clear that I I think I’d
click here…
the next round it’s likely that all three users will uncover a new set of problems,
> We’re all beginners under the skin. Scratch an expert and you’ll often find click here…
videotape and they’re very easy to share over a network. I recommend that you value their opinion, and (b) people tend to show up on time, eager to since they won’t be getting stuck on the first set of problems.
someone who’s muddling through—just at a higher level.
always use a screen recorder during user tests.
TWO TESTS WITH 3 USERS
TOTAL PROBLEMS
participate. Remember, even if the session is only 30 minutes, people usually Testing only three or four users also makes it possible to test and debrief in the FOUND: 9
> It’s usually not a good idea to design a site so that only your target have to block out another hour for travel time. Also, I’d rather have people who same day, so you can take advantage of what you’ve learned right away. Also, First test: 3 users
Second test: 3 users
audience can use it. If you design a site for accountants using terminology are curious about the process than people who are desperate for the money.
Who should do the testing?
Three users may
But in the
when you test more than four at a time, you usually end up with more notes than that you think all accountants will understand, what you’ll probably discover not find as many
second test,
> Keep the invitation simple. “We need to have a few people look at our Web Almost anyone can facilitate a usability test; all it really takes is the courage to try anyone has time to process—many of them about things that are really “nits,”
problems in a
with the first
is that a small but not insignificant number of accountants won’t know what site and give us some feedback. It’s very easy, and would take about forty-five it. With a little practice, most people can get quite good at it.
which can actually make it harder to see the forest for the trees.
single test.
set of problems
you’re talking about. And in most cases, you need to be addressing novices as fixed, they’ll
minutes to an hour. And you’ll be paid $___ for your time.”
well as experts anyway, and if your grandmother can use it, an expert can.
Try to choose someone who tends to be patient, calm, empathetic, a good listener, In fact this is one of the reasons why I’ve almost completely stopped generating find problems
they couldn’t
> Avoid discussing the site (or the organization behind the site) and inherently fair. Don’t choose someone whom you would describe as written reports (what I refer to as the “big honking report”) for my expert
> Experts are rarely insulted by something that is clear enough for have seen in
beforehand. You want their first look to tell you whether they can figure out
“definitely not a people person” or “the office crank.”
reviews and for usability tests. I finally realized that for most Web teams their the first test.
beginners. Everybody appreciates clarity. (True clarity, that is, and not just Test subject (A) sits in front of computer monitor (B),
Meanwhile, cable (F) carries signal from
what it is from a standing start. (Of course, if they’re coming to your office, ability to find problems greatly exceeds the resources they have available to fix something that’s been “dumbed down.”)
while facilitator (C) tells him what to do and asks ques-
camcorder to TV (G) in a nearby room where
they’ll have a pretty good idea whose site it is.)
tions. Camcorder (D) powered by squirrel (E) is pointed
interested team members (H) can observe.
them, so it’s important to stay focused on the most serious problems. Instead of Who should observe?
The exceptions:
at the monitor to record what the subject sees.
written reports, nowadays I report my findings in a conference call with the
> Don’t be embarrassed to ask friends and neighbors. You don’t have to feel Anybody who wants to. It’s a good idea to encourage everyone—team members, entire Web team, which may last for an hour or two. By the end of the call, we’ve Recruit loosely and grade on a curve
> If your site is going to be used almost exclusively by one type of user and like you’re imposing if you ask friends or neighbors to participate. Most people You can use the video cable to run the signal from the camcorder to a TV in people from marketing and business development, and any other stakeholders—
all agreed which problems are most important to fix, and how they’re going to fix it’s no harder to recruit from that group, then do it. For instance, if your enjoy the experience. It’s fun to have someone take your opinion seriously and another office—or even a cubicle—nearby so everyone on the development team When people decide to test, they often spend a lot of time trying to recruit users to attend.
them.
audience will be almost entirely women, then by all means test just women.
get paid for it, and they often learn something useful that they didn’t know can watch without disturbing the user.
who they think will precisely reflect their target audience—for instance, male about the Web or computers in general.
When people ask me how they can convince senior management that their accountants between the ages of 25 and 30 with one to three years of computer
> If your audience is split between clearly defined groups with very The camcorder needs to transmit what the user sees (the computer screen or the organization should be investing in usability, my strongest recommendation experience who have recently purchased expensive shoes.
divergent interests and needs, then you need to test users from each group designs on paper, depending on what you’re testing) and what the user and the doesn’t have anything to do with things like “demonstrating return on at least once. For instance, if you’re building a university site, for at least one facilitator say. In a usability lab, you’ll often see a second camera used to show the The best-kept secret of usability testing is the extent to which it doesn’t much round of testing you want to recruit two students, two professors, two high observers the user’s face, but this isn’t necessary: The user’s tone of voice usually matter who you test.
4
2
There are a number of screen recorders available, but I’m partial to Camtasia, made See Jakob Nielsen’s March 2000 Alertbox column “Why You Only Need to Test with 5 Users”
school seniors, and two administrators. But for the other rounds, you can conveys frustration pretty effectively.
by TechSmith, the same company that makes the screen capture program SnagIt at www.useit.com for a good discussion of the topic.
For most sites, all you really need are people who have used the Web enough to choose any mix.
You can buy the camcorder, TV, cable, and tripod for less than $600. But if your (http://www.techsmith.com). It’s very reliable and has a number of extremely useful 3 If you’re hiring someone to do the testing for you and money is no object, you might as well know the basics.
budget won’t stretch that far, you can probably twist somebody’s arm to bring in features, and it costs
about $300. For $1,000 more, they have a product called Morae test six or eight users since the additional cost per user will be comparatively low. But only if specifically designed for capturing usability tests—sort of like Camtasia on steroids—which a camcorder from home on test days.
it won’t mean you’ll do fewer rounds of testing.
allows observers to view the test live on a networked PC, eliminating the need for a camcorder.
It was very gratifying how many people
[ 138 ]
[ 139 ]
[ 140 ]
[ 141 ]
[ 142 ]
[ 143 ]
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
u s a b i l i t y t e s t i n g o n 10 c e n t s a d ay c h a p t e r 9
As a rule, you’ll always get more revealing results if you can find a way to observe users doing tasks that they have a hand in choosing. It’s much better, INTRODUCTION
If you have questions, just ask. I may not be
BACKGROUND QUESTIONS
It’s important to mention
Do you have any favorite Web sites?
REACTIONS TO THE HOME PAGE
for instance, to say “Find a book you want to buy, or a book you bought this, because it will seem
able to answer them right away, since we’re
Hi, Janice. My name is Steve Krug, and
This whole first section is
rude not to answer their
recently” than “Find a cookbook for under $14.” When people are doing made-interested in how people do when they don’t
Yahoo, I guess. I like Yahoo, and I use it all
First, I’m just going to ask you to look at this
The browser has been open,
the script that I use when I
I’m going to be walking you through
questions as you go along.
Before we look at the site, I’d like to ask you
up tasks, they have no emotional investment in it, and they can’t use as much but minimized. At this
conduct tests.5
have someone sitting next to them, but I will
You have to make it clear
I find it’s good to start
the time. And something called
page and tell me what you think it is, what
this session.
just a few quick questions. First, what’s your
with a few questions to get
point, I reach over and
of their personal knowledge.
before you start that (a)
try to answer any questions you still have
Snakes.com, because I have a pet snake.
strikes you about it, and what you think you
click to maximize it.
it’s nothing personal, and
occupation?
a feel for who they are and
when we’re done.
how they use the Internet.
would click on first.
As you begin designing your own site, it’s never too early to start showing your (b) you’ll try to answer
them at the end if they still
It gives them a chance to
Really? What kind of snake?
Don’t be afraid to digress
design ideas to users, beginning with your first rough sketches. Designers are You probably already know, but let me explain
I always have a copy in
We have a lot to do, and I’m going to try to
want to know.
I’m a router.
loosen up a little and gives
and find out a little more
front of me, and I don’t
For now, don’t actually click on anything. Just
wrote to tell me that they started doing their
often reluctant to show work in progress, but users may actually feel freer to why we’ve asked you to come here today. We’re
keep us moving, but we’ll try to make sure that
you a chance to show that
about the user, as long as
hesitate to read from it,
you’re going to be listening
A python. He’s about four feet long, but
tell me what you would click on.
comment on something that looks unfinished, since they know you haven’t got testing a Web site that we’re working on so we
you come back to the topic
but I find it’s good to ad lib
it’s fun, too.
I’ve never heard of that before. What does a
attentively to what they
he should get to be eight or nine when
before long.
as much invested in it and it’s still subject to change. Also, since it’s not a polished can see what it’s like for actual people to use it.
a little, even if it means
router do, exactly?
say—and that there are no
You may have noticed the camera. With your
At this point, most people
he’s fully grown.
And again, as much as possible, it will help us
design, users won’t be distracted by details of implementation and they can focus making mistakes. When the
wrong or right answers.
I want to make it clear right away that we’re
users see that I’m
will say something like, “I’m
permission, we’re going to record the computer
if you can try to think out loud so we know
on the essence and the wording.
comfortable making
not going to end up on
Not much. I take orders as they come in,
testing the site, not you. You can’t do anything
screen and what you have to say. The recording
Don’t hesitate to admit
Wow. OK, now, finally, have you bought
what you’re thinking about.
mistakes, it helps take the
America’s Funniest Home
and send them to the right office.
Later, as you begin building parts of the site or functioning prototypes, you can wrong here. In fact, this is probably the one
your ignorance about
pressure o◊ them.
will be used only to help us figure out how to
Videos, am I?”
anything on the Internet? How do you feel
anything. Your role here
begin testing key tasks on your own site.
place today where you don’t have to worry
improve the site, and it won’t be seen by anyone
about buying things on the Internet?
Good. Now, roughly how many hours a week
is not to come across
about making mistakes.
except the people working on the project. It
as an expert, but as a
I also recommend doing what I call Cubicle tests: Whenever you build a new would you say you spend using the Internet,
also helps me, because I don’t have to take as
good listener.
I’ve bought some things recently. I didn’t
kind of page—particularly forms—you should print the page out and show it to We want to hear exactly what you think, so
including email?
many notes. There are also some people
do it for a long time, but only because I
the person in the next cubicle and see if they can make sense out of it. This kind please don’t worry that you’re going to hurt
watching the screen in another room.
couldn’t get things delivered. It was hard
of informal testing can be very efficient, and eliminate a lot of potential problems.
our feelings.6 We want to improve it, so we
Oh, I do
n’t know. Probably an hour a day at
Notice that she’s not sure
how much time she really
to get things delivered, because I’m not
need to know honestly what you think.
If you would, I’m going to ask you to sign
Give them the release and
work, and maybe four hours a week at
spends on the Internet.
A sample test session
non-disclosure agreement
home during the day. But now one of my
something for us. It simply says that we have
home. Mostly that’s on the weekend. I’m
Most people aren’t. Don’t
As we go along, I’m going to ask you to think
(if required) to sign. Both
neighbors is home all the time, so I can.
your permission to record you, but that it will
too tired at night to bother. But I like
worry. Accurate answers
Here’s an annotated excerpt from a typical—but imaginary—test session. The site out loud, to tell me what’s going through your
should be as short as
aren’t important here. The
only be seen by the people working on the
possible and written in
playing games sometimes.
is real, but it has since been redesigned. The participant’s name is Janice, and mind. This will help us.
7
main point here is just to
plain English.
And what have you bought?
project. It also says that you won’t talk to
get her talking and
she’s about 25 years old.
anybody about what we’re showing you today,
How do you spend that time? In a typical day,
thinking about how she
uses the Internet and to
Well, I ordered a raincoat from L.L. Bean,
since it hasn’t been made public yet.
for instance, tell me what you do, at work and
give you a chance to gauge
and it worked out much better than I
at home.
what kind of user she is.
Do you have any questions before we begin?
thought it would. It was actually pretty easy.
Well, at the office I spend most of my time
No. I don’t think so.
OK, great. We’re done with the questions, and
checking email. I get a lot of email, and a
we can start looking at things.
lot of it’s junk but I have to go through it