by Steve Krug
You have to be prepared to be persistent and a bit ruthless. It may feel like you’re being rude, but remember that you’re paying them for their time, and if you don’t get what you need, you’re wasting your time, their time, and your observers’ time.
Even after you get them back on track, some will relapse. Bite your tongue but be patient. Someone who seems hopeless may come around and end up providing you with really valuable insights.
In extreme cases, if you’re not getting any benefit from the participant, you may decide to end the session early. For instance, you may have someone who is clearly not qualified. Either your recruiting wasn’t up to snuff, or they were deceptive when you spoke to them.
If you feel the need to end the session early, you can use any plausible (and hopefully convincing) excuse, thank them, pay them, and get ready for your next session.
10 Do yourself a favor and search the Web for an audio fi le of Bob and Ray’s “Slow Talkers of America” sketch.
[ 86 ]
mind reading made easy
Don’t worry, be happy
Reading about all of this may make the facilitator’s job sound like a lot to handle, but the truth is almost everyone who tries it finds it to be surprisingly easy. Most people get the hang of it very quickly, but it’s only natural to be a little anxious (or for some people, very anxious) the first few times you run a test. Here are two things that can help minimize any stage fright: Practice reading the script aloud. First read it out loud with no one around four or five times, then read it to one or two people: a family member, for instance, or co-workers. By then you won’t be self-conscious about it anymore.
Do a practice test with no pressure. If you find you’re really anxious about your first public test, try doing a “dry run.” Get two friends to be the participant and an observer and do everything you would in a real test, including setting up screen sharing in another room.
FAQ
Who should be a facilitator?
Probably you. You’ve demonstrated interest by reading this book, and interest is the best qualifier. It certainly helps if you’re a good listener and you’re comfortable chatting with strangers. But as someone once pointed out to me, you don’t have to actually be a “people person” to facilitate well, as long as you can pretend to like people.
As time goes on, you’ll probably want to train someone else on your team to facilitate, too, so you can just observe and take notes. (As the person most interested in usability, your notes and observations are usually the most valuable.)
Who shouldn’t be a facilitator?
Anyone who really doesn’t like people—the office curmudgeon, for
instance—is probably a poor choice. Also people who don’t listen well, people who tend to be impatient, and people who like to force their ideas on others.
[ 87 ]
chapter 8
The worst choice would be someone who has a personal agenda about the right way to design things that they can’t put aside.
Where should I sit? Next to the participant? Behind him?
The participant needs to sit directly in front of the monitor and keyboard, and you need to be positioned so you can see the screen clearly enough to follow what they’re doing. I find that it works best if I’m sitting next to the participant and slightly behind, just far enough away so they won’t feel like I’m hovering.
Should I take notes while I’m facilitating?
As you gain experience, you’ll find that you can take some notes and pay full attention to the participant and keep the session moving, all at the same time. In the beginning, though, I recommend that most of the notes you take during the tests should just be reminders about things that you want to follow up on during the probing, like “Did he see download link?”
The observers will be taking plenty of notes, and if you need to you can always go back and look at the recording. But don’t forget to make your Top Problems list after each session while it’s still fresh in your mind.
Why don’t you ask more entrance and exit questions?
Pre- and post-test questions are often used to try to assess things like whether people find the site usable and whether using the site improves their opinion of your organization or product. These can be very valuable and they certainly make marketing people happy, but I don’t think they have a place in do-it-yourself testing.
For one thing, the samples are too small to have any meaning. And there’s also the problem that people are notoriously bad at this kind of self-reporting. The biggest running joke among usability professionals is that we’ve all seen people who have struggled almost to the point of tears trying to use a system that just doesn’t work the way it should. But when it comes time to rate it on a scale of 1 (user-hostile) to 7 (extraordinarily user-friendly), they’ll give it a “6.” We don’t know why it happens, but it does.
All the time.11
11 It may be because they think of you as their host—you’ve been nice to them and you’re paying them for their time so they don’t want to seem rude. Or it may just be that people
[ 88 ]
mind reading made easy
You recommended Camtasia. What about Morae?
Some years ago, so many people were using Camtasia to record
usability tests that the folks at TechSmith decided to build another product specifically designed for usability testing: Morae. I think of it as Camtasia on steroids. It has a ton of additional features, including a logging capability which makes it easy for an observer to take notes that are synched to the recording. And it has its own remote viewer which eliminates the need for a separate screen sharing solution.
It’s a wonderful tool and a lot of people use it, but for the kind of testing I’m talking about, I think it’s overkill for most people. I’d recommend starting out with the simpler tool and graduating to Morae when you have a need for it. In the meantime, you may want to download the 30-day free trial and learn what it can do.
What about recording the user’s face?
I’ve never been a big fan of recording the user’s face during testing.
The original purpose of this picture-in-
picture feature (or perhaps more accurately,
“pain-in-picture”) was to capture the
participant’s frustration to use as proof that
the product needed more usability work. But
I think this “SquirmCam” can actually be an
unnecessary distraction.
If you have good quality audio, observers can almost always tell what the user is feeling from their tone of voice.
have such low expectations that your site seems no worse than most. Personally, I think it’s a variant of Stockholm syndrome, where hostages develop an emotional bond with their captors, sympathizing with them and even defending them after they’re fi nally freed.
[ 89 ]
chapter
chapter9
9
Make it a
spectator sport
getting everyone to watch and telling them
what to look for
[ 90 ]
You can observe a lot by watching.
—LAWRENCE “YOGI” BERRA
O ne of the most valuable pieces of advice I can give you about usability testing is to do everything you can to get as many people in your organization as possible (stakeholders, managers, developers, designers, editors, writers—
even executives) to attend your test sessions in person. Or, in maxim-speak: Make it a
spectator sport.
Why do I think it’s so important to get people there in person? Because, when it comes to usability testing...
Seeing is believing
It’s another one of the things that everyone who’s done a lot of testing knows: watching usability tests in person is a transformative experience. People often go into their first test with some skepticism, but they almost invariably come out...changed.
The most obv
ious change is that they’re not skeptical about testing anymore; in fact, they’re usually enthusiastic about it. It’s hard to watch tests and not understand that what you’re seeing is very valuable.
But there’s a subtler and more significant change: watching usability testing makes you realize that your users aren’t just like you. Most people think that all users are just like them when it comes to using the Web. Watching real users gives them that eureka moment: they’re not all like me, and in fact they’re not all like anybody. I like to say that watching usability tests is like travel: it’s a broadening experience. You realize that the rest of the world
[ 91 ]
chapter 9
doesn’t live and think the same way you do. This profoundly and permanently changes your relationship to users, making you a better developer, designer, manager, or whatever you are.
For reasons that aren’t entirely clear, this transformative effect is much more pronounced when you watch tests live and in person than when you watch clips or a recorded session. It’s like the difference between watching a sporting event live on television and watching a replay later: “live” is just more compelling. And when you attend a session with others, you also benefit from the shared group experience and the opportunity to compare observations during and between test sessions.
Whatever the reason is, believe me: it pays to get people in the room.
The more, the merrier
When you start doing monthly testing, the people who are directly involved in the part of the site that’s being tested will probably be eager to come and watch.
But you want to make a point of inviting and encouraging everyone to attend: designers, developers, product managers, bosses, marketing people, writers, editors, and all the various stakeholders who have interest or influence in the design and content.
[ 92 ]
make it a spectator sport
Do everything you can to get people to come. A few things that tend to work: Make it easy for people to attend. It helps to schedule your monthly usability testing on a “slow” day of the week and at a slow time of the month.
Advertise. Send out a “save the date” email two weeks before the test day that says what you’ll be testing; then send out a tickler email a few days before the tests and a last-minute reminder the day before.
Make it clear what’s in it for them. Stakeholders are always interested in getting their “pet” problems fixed. Make sure they understand that attending tests gives them a voice in the debriefing where these things are decided.
Trick executives into coming. I always tell people to do whatever it takes to get people from management to attend. Tell them that it will be good for morale if they could just drop by for a bit. I’ve seen VPs who
“dropped by” cancel meetings so they can stay and keep watching. Dilbert notwithstanding, these are usually smart people who recognize the value of this kind of input once they see it firsthand.
Provide quality snacks. Word will get around.
What do observers do?
The observers’ job is very simple:
Watch and learn, and take some notes.
At the end of each session, write down the three most important usability problems they saw in that session.
Suggest questions they’d like to have the facilitator ask the participant.
Enjoy
the snacks.
Come to the lunchtime debriefing session.
That’s it. Here’s a set of instructions you should give them.1
1 You can download this handout from the book Web site and edit it (if you’re doing longer or shorter sessions, for instance).
[ 93 ]
chapter 9
Instructions for Usability Test Observers
Thanks for coming to today’s tests. Each of the three sessions will last about 50
minutes, with a 10-minute break in between.
To get as much as we can out of these tests, we need your help with a few things:
Take notes. Please make notes about anything interesting you notice, particularly points where the user was confused or couldn’t get the tasks done. We’ll be comparing notes during the debriefing session at lunchtime today.
Make a list at the end of each session. During the break between sessions, use the attached sheet to jot down the three most serious usability problems you noticed in that session.
Come to the debriefing. (Free lunch!) If at all possible, we’d love to have you join us at ___ pm in room _____________, where we’ll compare notes and decide which usability problems we’re going to fix in the next month.
If you think of a question you’d like to ask the participant, write it down. Near the end of each session, we’ll check to see if you have any questions.
Stay as long as you can. We know you have other commitments, but there are only a few sessions, and each one will offer different lessons.
Even if you start to lose interest, try to keep watching and listening—you never know when the participant will say something revealing. You can come and go if you need to, but please try to do it unobtrusively
Try to avoid distracting others. Following a test can require concentration.
Try to limit your conversation to what you’re observing. If you need to have another kind of discussion or answer a phone call, please step outside the room. Think of it as a movie theater: don’t talk loud enough or long enough that the people around you can’t follow the plot.
Thanks for your help!
[ 94 ]
make it a spectator sport
Top Three Usability Problems
After each test session, list the three most serious usability problems you noticed.
Participant #1
1. ................................................................................................................................
2. ................................................................................................................................
3. ................................................................................................................................
Participant #2
1. ................................................................................................................................
2. ................................................................................................................................
3. ................................................................................................................................
Participant #3
1. ................................................................................................................................
2. ................................................................................................................................
3. ................................................................................................................................
[ 95 ]
chapter 9
The observation room
A conference room is usually ideal for the observers. If you outgrow the conference room, you can use a training room or a small auditorium—
anywhere that people can see the screen and hear the audio.
One important consideration: the observation room and the test room should not be right next to each other. You don’t want the participants to hear group laughter (or collective groans) that are in synch with what they’ve just done.
Very bad.
From test room
(page 65)
So, what would
I think I’d click
you do next?
here….
c
b
a
I didn’t see
d
that coming.
e
a Computer b Projector c Speakers d Snacks e Speakerphone
[ 96 ]
make it a
spectator sport
a) A computer with Internet access and screen sharing client software.
The computer can be a laptop or desktop, PC or Mac, and you’ll need
Internet access for screen sharing. For some kinds of screen sharing software you’ll have to install a viewer, but many—including GoToMeeting
—use a Web browser, so no installation is necessary.
b) The image from the projector (or large screen monitor) needs to be large enough and bright enough so everyone can follow what the participant is doing. This is less crucial if you’re testing something the observers are very familiar with, but if you’re testing new designs, competitors’ sites, or pages with dynamic content, observers will need to be able to see the screen in some detail.
People sitting farthest from the screen will often find it easier to watch the session on their laptops, but you have to be wary of people drifting off into the world of email.
c) A pair of powered speakers. For the same reason you
want a good microphone in the test room, you want good
speakers for the observers. I recommend the Logitech
X-140 powered speakers, which cost about $25. They’re
quite clear and loud and have their own volume control.
d) Snacks. One excellent way to make the observation
room pleasant and inviting so people will want
to come back is to provide food. Don’t scrimp on
snacks! Think of them as a lure: What kind of food
is most likely to attract the Web team at 9 a.m.?
Bagels and muffins are usually a good bet, but you
should follow the local customs. If your team
is partial to granola bars and Twizzlers, give them
granola bars and Twizzlers.
e) A
speakerphone.
You should have a speakerphone available as a backup,
and make sure that you have the phone number for the test room.
[ 97 ]
chapter 9
Appoint a hall monitor
Since you’re going to be busy in the test room, it’s highly advisable to have someone minding the store for you in the observation room. Just ask someone who’s planning on attending anyway and who won’t mind pitching in—