by Steve Krug
Remote testing requires more concentration, and not having the ability to
“read” the person visually is a much more significant loss for a beginner.
I’d recommend that you wait until you’ve done about three monthly rounds of testing before you start doing them remotely. By then, the whole process will feel much more routine and you’ll be more relaxed and better able to cope with the unexpected.
(Of course, if you want, you can begin experimenting with remote testing before then. I just wouldn’t do it for your public sessions.)
Should we still have an observation room?
Yes. It’s just as important to have people observe remote tests as it is for in-house tests. You want that “clubhouse” effect where people compare notes and share the experience. Since they’ll be observing via screen sharing in either case, from their perspective the experience will be exactly the same.
[ 139 ]
chapter
chapter15
15
Overachievers
only
recommended reading
[ 140 ]
Isn’t there somebody a little more qualified?
— BOBCAT GOLDTHWAIT, WHEN ASKED IF HE WANTED TO
CUT HIS NEWBORN SON’S UMBILICAL CORD
Once you’ve started doing some testing, some of you will want to learn more about it.1 For the benefit of you overachievers, here are my favorite books about testing and related topics.
Books about testing in general
As I said in the introduction, I deliberately haven’t tried to cover every aspect of testing in detail. These three books do just that, and they do it very well; you can’t go wrong with any one of them.
Handbook of Usability Testing (Second Edition)
Jeffrey Rubin and Dana Chisnell, John Wiley & Sons, 2008.
Jeff Rubin’s book was long one of the best on the subject, and
the new edition with co-author Dana Chisnell has made a very
good thing even better.
A Practical Guide to Usability Testing (Revised Edition)
Joseph Dumas and Janice (Ginny) Redish, Intellect, 1999.
Between them, Joe and Ginny probably know more about
usability testing than the rest of us put together, and they both make a wonderful habit of sharing what they know.
1 Some of you won’t, which is fi ne, too. Personally, I’ve tried for years now to convince people that I’m really not one of those people who have limitless curiosity about things, but I’ve had only modest success.
[ 141 ]
chapter 15
Usability Testing Essentials: Ready, Set, Test!
Carol Barnum, Longman, 2010.
As I write this, Carol is still working on this major revision
of her excellent 2002 book, but I know it will be well worth
reading, with new topics like accessibility and international
usability testing.
Specific topics
Paper Prototyping
Carolyn Snyder, Morgan Kaufmann, 2003.
As Johnny Carson would have said, “Every single thing you need to know about paper prototyping is in this book.” And a
very good book on testing in general, too.
Moderating Usability Tests
Joseph Dumas and Beth Loring, Morgan Kaufmann, 2008.
An entire book—based on 40 years of combined experience—
about the process of facilitating a test. A very quick and very
informative read.
Measuring the User Experience
Thomas Tullis and William Albert, Morgan Kaufmann, 2008.
If you need to do some quantitative testing (for instance, if
your boss insists on a benchmark test so you can “prove” your
site has improved later), you must read this book.
[ 142 ]
overachievers only
Recruiting Without Fear
Will Schroeder, David Brittan, and Jared Spool. Usability Interface Engineering, 43-page downloadable PDF, $49.99
http://www.uie.com/reports
Jared Spool’s company has been recruiting test participants
since 1988, and this white paper explains how they do it.
233 Tips and Tricks for Recruiting Users as Participants
in Usability Studies
Deborah Sova and Jakob Nielsen, Nielsen Norman Group,
144-page downloadable PDF, $79.95
http://www.nngroup.com/reports
Co-author Deborah Sova draws on years of experience as a
recruiter to offer plenty of sound advice.
Books about fixing things
Letting Go of the Words: Writing Web Content
That Works
Janice (Ginny) Redish, Morgan Kaufmann, 2007.
Ginny’s book is the best advice available on fixing usability problems that are the result of less-than-perfect writing or
editing—and avoiding them in the first place. One Web writer
described it to me as “life-altering” and I think she’s right.
Forms that Work: Designing Web Forms for Usability
Caroline Jarrett and Gerry Gaffney, Morgan Kaufmann, 2008.
Almost every Web site has some forms, and almost every
Web form has usability problems. This book is to forms what
Ginny’s book is to writing.
[ 143 ]
opening remarks
chapter
16
Happy trails /
to you
a few final words of encouragement
[ 144 ]
Happy trails / to you
until / we meet / again
—ROY ROGERS AND DALE EVANS
Here are all of my “maxims.”
A morning a month,
that’s all we ask.
Start earlier than you
think makes sense.
Recruit loosely and
grade on a curve.
Make it a spectator sport.
Focus ruthlessly on a small number
of the most important problems.
When fixing problems, always do
the least you can do.
Keep these in mind, and you’ll do fine. And remember, everything in here is just a recommendation. Feel free to experiment and do whatever works for you.
Good luck, and please let me know how you make out. (You can write to me at [email protected].)
[ 145 ]
sample test script and consent form
Sample test
script and
consent form
[ 146 ]
Test script
�Web browser should be open to Google or
some other “neutral” page
Hi, _____________. My name is ___________, and I’m going to be
walking you through this session today.
Before we begin, I have some information for you, and I’m
going to read it to make sure that I cover everything.
You probably already have a good idea of why we asked
you here, but let me go over it again briefl y. We’re asking
people to try using a Web site that we’re working on so we
can see whether it works as intended. The session should
take about an hour.
The fi rst thing I want to make clear is that we’re testing
the site, not you. You can’t do anything wrong here. In fact, this is probably the one place today where you don’t have
to worry about making mistakes.
As you use the site, I’m going to ask you as much as
possible to try to think out loud: to say what you’re looking
at, what you’re trying to do, and what you’re thinking. This
will be a big help to us.
[ 147 ]
sample test script and consent form
Also, please don’t worry that you’re going
to hurt our
feelings. We’re doing this to improve the site, so we need to
hear your honest reactions.
If you have any questions as we go along, just ask them.
I may not be able to answer them right away, since we’re
interested in how people do when they don’t have someone
sitting next to them to help. But if you still have any
questions when we’re done, I’ll try to answer them then.
And if you need to take a break at any point, just let me
know.
You may have noticed the microphone. With your
permission, we’re going to record what happens on the
screen and our conversation. The recording will only be
used to help us fi gure out how to improve the site, and it
won’t be seen by anyone except the people working on this
project. And it helps me because I don’t have to take as
many notes.
Also, there are a few people from the Web design team
observing this session in another room. (They can’t see us,
just the screen.)
If you would, I’m going to ask you to sign a simple
permission form for us. It just says that we have your
permission to record you and that the recording will only
be seen by the people working on the project.
[ 148 ]
sample test script and consent form
�Give them a recording permission form
and a pen
�While they sign it, START the SCREEN
RECORDER
IF YOU ARE USING A NONDISCLOSURE AGREEMENT (optional):
I know we also sent you a nondisclosure agreement
that says that you won’t talk to anybody about what
we’re showing you today, since it hasn’t been made
public yet. Do you have that with you?
�Accept the NDA and make sure that it’s signed.
If they don’t have it with them, hand them a copy
and give them time to read and sign it.
Do you have any questions so far?
OK. Before we look at the site, I’d like to ask you just a few
quick questions.
First, what’s your occupation? What do you do all day?
[ 149 ]
sample test script and consent form
Now, roughly how many hours a week altogether—just
a ballpark estimate—would you say you spend using the
Internet, including Web browsing and email, at work and
at home?
And what’s the split between email and browsing—a
rough percentage?
What kinds of sites are you looking at when you browse
the Web?
Do you have any favorite Web sites?
OK, great. We’re done with the questions, and we can start
looking at things.
�Click on the bookmark for the site’s
Home page.
First, I’m going to ask you to look at this page and tell me
what you make of it: what strikes you about it, whose site
you think it is, what you can do here, and what it’s for. Just
look around and do a little narrative.
[ 150 ]
sample test script and consent form
You can scroll if you want to, but don’t click on anything
yet.
�Allow this to continue for three or four minutes,
at most.
Thanks. Now I’m going to ask you to try doing some
specifi c tasks. I’m going to read each one out loud and give
you a printed copy.
I’m also going to ask you to do these tasks without using
Search. We’ll learn a lot more about how well the site
works that way.
And again, as much as possible, it will help us if you can
try to think out loud as you go along.
�Hand the participant the fi rst scenario, and read
it aloud.
�Allow the user to proceed until you don’t feel like
it’s producing any value or the user becomes very
frustrated.
�Repeat for each task or until time runs out.
[ 151 ]
sample test script and consent form
Thanks, that was very helpful.
If you’ll excuse me for a minute, I’m just going to see if the
people on the team have any follow-up questions they’d
like me to ask you.
�Call the observation room to see if the observers
have any questions.
�Ask the observers’ question(s) and then probe
about anything you want to follow up on.
Do you have any questions for me, now that we’re done?
�Give them their incentive, or remind them
it will be sent to them.
�Stop the screen recorder and save the fi le.
�Thank them and escort them out.
[ 152 ]
sample test script and consent form
Recording consent form
Thank you for participating in our usability research.
We will be recording your session to allow
[ORGANIZATION NAME] staff members who are
unable to be here today to observe your session and
benefi t from your comments.
Please read the statement below and sign where
indicated.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
I understand that my usability test session will be
recorded.
I grant [ORGANIZATION NAME] permission to use
this recording for the purpose of improving the designs
being tested.
Signature: ___________________________________________
Print your name: ___________________________________
Date: __________________________________________________
[ 153 ]
acknowledgments
Acknowledgments
[ 154 ]
I have always depended on the kindness of strangers.
—BLANCHE DUBOIS IN A STREETCAR NAMED DESIRE
T he people involved in getting this book done weren’t strangers to me: I was lucky enough to be able to round up the same team that made Don’t Make Me Think happen. But I have relied deeply on their kindness and their extraordinary patience and goodwill in the face of my writing habits.
In no particular order:
My reviewers—Joe Dumas, Caroline Jarrett, Karen Whitehouse, and Paul Shakespear—who all spent precious time to keep me from appearing foolish. To protect the innocent, I feel compelled to note that inclusion in this list does not imply agreement with everything in the book.
Elisabeth Bayle. Before Elisabeth appeared three years ago, I’d worked alone for almost 30 years and could never imagine it otherwise (largely the result of a nightmarish collaboration in the early ’80s). Since then, I’ve had the pleasure of a colleague and a friend who knows as much about this stuff as I do. I stretch her patience constantly and we sometimes disagree, but we have a rule about not throwing things.
Barbara Flanagan, copyeditor, old friend, and grammar maven, without whom this book would have a copyright date of 2014. Any instances of things like “who” where it should be “whom” are attributable to my stubbornness and her indulgence. I would love to write a book with her about how to write.
Allison Cecil (and her Great Danes), who took time out from flattening and hand stamping 4,000 pieces of silverware into
beautiful garden markers (available at Anthropologie) to design
yet another book for me.
[ 155 ]
acknowledgments
Mark Matcho, whose illustrations add so much.
Ginny Redish and Caroline Jarrett for being themselves.
The large community of usability professionals, who tend to be a very nice bunch of folks. Go to an annual UPA conference and find out for yourself.
Randolph Bias and Carol Barnum, who both understand the theoretical underpinnings of this far better than I ever will and were brave enough to do a panel with me at the 2008 UPA conference titled “Discount Testing by Amateurs: Threat or Menace?”
My friends Richard Gingras and Mitzi Trumbo, who were so patient with a houseguest who was mostly stuck to his computer, writing, even in the face of a cliffside view of the Pacific.
Harry, now in college, who still manages to send me the occasional link to things that he knows will make me laugh.
And finally, Melanie, for being supportive even when she insisted she wasn’t.
As Richard Fariña said in one of the “Little Nothing Poems” he wrote for Mimi, “Nothing matters / any more.” 1
1 Just in case it’s not clear (and I have to admit that poetry is often not clear to me ), there’s an implied “than you” at the end:
Nothing matters
any more [than you].
[ 156 ]
This page intentionally left blank
index
Index
[ 158 ]
index
A
F
A/B testing, 119
facilitator
a morning a month, 23
role of, 63
acknowledgment tokens, 84
similarities to therapist, 63, 81–85
Agile development, 27
amateurs, usability testing by, 6
G
B
Google Website Optimizer, 119
GoToMeeting, 66, 137
budget for testing, 28
grade on a curve, 42, 74
C
H
Camtasia, 66, 89
hall monitor, 98
checklists, 57–61, 68
instructions for, 99
consent form, recording, 153
hallway tests, 118
Home page tour, 75
hurt feelings, dealing with, 100
D
debriefing meeting
I
how to run, 105
purpose of, 103
incentives for participants, 48