by Steve Krug
preferably someone who isn’t easily intimidated by co-workers and bosses.
Here’s a page of instructions you can give to your hall monitor:
[ 98 ]
make it a spectator sport
Hall Monitor’s Guide
Thanks for helping out with today’s usability tests!
Since I’ll be in the test room with the participants, I need your help making sure things run smoothly in the observation room.
Here’s what you can do:
Read
the
Instructions for Usability Test Observers so you know what observers need to do.
Make sure that everyone gets a copy of the handouts as they arrive: Instructions for Usability Test Observers
The test script
The scenarios for the tasks the participants will be doing
Make sure everyone can see and hear the test. If there’s a problem with the screen sharing or the audio, try to troubleshoot it. If you can’t get it working right away, call me in the test room at ________. I’ll stop the test and help you fix it.
Try to head off any extended off-topic conversations, which can interfere with people’s ability to concentrate on the test. (Limited conversation about what’s happening in the test room is fine.)
Remind people to step outside if they need to take phone calls. (Usually all you have to do is make eye contact with them and point to the door—with a smile, of course—as they put the phone to their ear.)
As soon as each session ends, remind everyone to go back through their notes and jot down the top three problems they noticed during the test.
And if they can’t come to the debriefing, ask them to leave their list of problems with you.
[ 99 ]
chapter 9
FAQ
Aren’t people’s “feelers” likely to get hurt?
People ask about this a lot: Isn’t it going to be painful for team members to watch people struggle—and possibly fail—while trying to use
something they had a hand in building? In front of their peers and perhaps their bosses, no less? Aren’t people likely to get defensive, disheartened, and even worried about losing their jobs when they see participants having trouble?
In my experience, it’s usually not a serious problem.
Watching the first round of testing of your site can be a bit of a shock, which is why I recommend doing your first tests on competitors’ sites, where the team members have nothing personal at stake and they can even indulge in a bit of harmless therapeutic schadenfreude.2
Most people realize quickly that even though testing exposes problems, more often than not it also suggests the solution—sometimes to a problem they’ve been wrestling with for a long time.
The only time I think team members are really troubled by testing is when it’s done so late in the development process that there’s no time to fix the problems that are uncovered. But you’re not going to do that.
If you feel that there are some bruised egos on your team after testing, you may want to go out of your way to say something at the debriefing to make them feel better, like “We saw some problems, but by and large, the thing worked remarkably well. And I think the problems we saw are quite fixable.”
Should people who can’t observe in person be allowed to view the sessions remotely via screen sharing?
It depends: is it that they can’t observe in person or that they don’t feel like it? People who really want to watch the sessions but are legitimately unable to (because they’re in another city, for instance) should certainly be allowed to. But I wouldn’t recommend giving people who are on-site the option of watching at their desks instead of coming to the observation 2 Pleasure derived from the misfortunes of others.
[ 100 ]
make it a spectator sport
room. For one thing, it means they’re not going to contribute to the group experience. More significantly, though, if they’re at their desk (and their computer), it’s going to be almost impossible for them to resist the siren song of multitasking, which means they’re not going to be paying enough attention to get much value out of “watching.”
Can observers be in the same room as the participant?
I don’t recommend it. Experienced facilitators can manage it if necessary, but it’s not a good idea for beginners.
Most participants are perfectly capable of ignoring an observer (or even two or three) in the same room with them. It’s the observers who are the problem. Inevitably, some people just can’t keep their mouths shut, and nothing can ruin a session like a manager who can’t refrain from asking marketing questions or a developer who’s dead set on getting the participant’s opinion about a new feature he’s thinking of adding.
If for some reason you have to have observers in the test room, introduce them to the participant and explain why they’re there. And make it crystal clear to the observers that they must follow some rules:
Be quiet. Turn off your cell phone and speak only when spoken to.
Don’t answer any questions the participant asks unless the facilitator specifically tells you to. Even then, keep your answer short and only answer the specific question that was asked.
Maintain a poker face. No frowning or smiling and no laughing unless the participant says something clearly meant to be funny. And above all, no sighing.
Don’t coach or help the participant in any way. No nodding or grinning when they do something right, for instance.
[ 101 ]
chapter
chapter10
10
Debriefing 101
comparing notes and deciding what to fix
[ 102 ]
There are some men here to flood the bed for skating.
— MR. MACGREGOR, ATTTEMPTING
TO WAKE ROBERT BENCHLEY1
The debriefing session has a very clearly defined purpose. You want to come out of the room with two things:
A list of the most serious usability problems that participants encountered while using your site.
A list of the problems you’re going to fix before next month’s round of testing.
The debriefing should take place as soon as possible after the test sessions while what happened is still fresh in everyone’s mind.
If possible, I recommend that you make it a rule that only people who have attended at least one of the morning’s test sessions can come to the debriefing.
It’s the price people have to pay to have a say in the meeting.
I realize that this may not be something you can enforce in your organization, and please don’t get yourself fired over it. But it really is one of the best ways to ensure that people will come to the test sessions. It also helps keep the debriefing focused on what people actually observed during the tests so it doesn’t devolve into “religious debates” based on personal opinions.
An hour is probably a good length for the meeting. Serve lunch, and don’t scrimp: get the good pizza.1
1 Robert Benchley was a seminal American humorist who wrote hundreds of columns for The New Yorker in the 1920s and ’30s (many subsequently published in collections with titles like David Copperfi eld, or Twenty Thousand Leagues Under the Sea ). If you laughed at something recently, it was probably in some way touched by Benchley’s legacy.
(Dave Barry has called Benchley “my idol,” for instance.) Mr. MacGregor was Benchley’s personal assistant, who was forced to resort to various ruses to get him out of bed in the morning.
[ 103 ]
chapter 10
Take the worst first
The most important thing you need to understand about fixing usability problems is that the following statements are all true:
1. All sites have usability problems.
2. All organizations have limited resources to devote to fixing usability problems.
3. You’ll always have more problems than you have the resources to fix.
4. It’s easy to get distracted by less s
erious problems that are easier to solve, which means the worst ones often persist.
Therefore:
5. You have to be intensely focused on fixing the most serious problems first.
And yes, it’s a maxim:
Focus ruthlessly on only the
most serious problems.
If you don’t follow this rule, I can almost guarantee that the worst usability problems will still be there a month from now. And six months from now.
How do you know which problems are the most serious?
Take my word for it: it’s usually pretty obvious which ones are the worst.
That’s one of the best things about usability testing. If you actually watch people use your site, you’ll know which problems are bad.
[ 104 ]
debriefing 101
There are basically two considerations that determine severity:
Will a lot of people experience this problem?
Will it cause a serious problem for the people who experience it, or is it just an inconvenience?
So, for instance, some problems are important to fix because even though they affect only a relatively small number of people, when they do they’re a source of real tsuris.2 (For example, the user might be unable to complete a transaction.)
Determining severity is always a judgment call. Problems that are going to cause a lot of people a lot of trouble are no-brainers. The toughest decisions involve corner cases (very damaging problems that affect only a few users) and ubiquitous nuisances (things that affect a lot of people but are really only minor annoyances).
How to run the debriefing meeting
The person running the tests (i.e., you) should run the debriefing. Here’s what you do:
1. Begin by explaining how the meeting is going to work.
“From your lists of usability problems that you observed during the test sessions we’re going to choose the ten most serious ones. Then we’re going to prioritize them and agree on which ones we’re going to commit to fixing in the next month.”
2. Ask everyone to review the list of problems they wrote down during the test sessions and choose the three that they think are the most serious.
3. Go around the room and ask people to read their three problems aloud.
(If they have one that’s already been mentioned, they can just say “I had ________, too.”)
2 Yiddish for distress, woe, or misery. A wonderful word.
[ 105 ]
chapter 10
4. Write them all down on an easel pad,3 taping sheets up on the wall as they get full. (Leave some room between items so you can add
variations suggested by others.)
5. When everyone has had a chance to contribute their three problems, look at the list and choose what seem to be the ten most serious.
You can ask people to vote if you want, but don’t be afraid to just say, “It sounded to me like these are the top ten” as you put checkmarks next to them. Then wait for any objections and make changes if necessary.
6. Write down a new rank-ordered list of these top ten problems, starting with the most serious. Leave some room between them where you’ll
make notes about how to fix them.
Again, use your own judgment about the order, but listen to any
reasonable suggestions about changes.
7. Working down the list without skipping any, have the team discuss briefly how each problem can be fixed within the next month. Try to keep the proposed fixes as simple as possible (see Chapter 11).
8. Continue working your way down the list until you feel like you’ve committed all the resources you have available for fixing things in the next month, then stop.
3 Feel free to use your list-making weapon of choice. But a whiteboard may turn out to be too small, and while a laptop connected to a projector has a lot of advantages it probably won’t allow everyone to see the entire list at once.
[ 106 ]
debriefing 101
Tips for success
Here are some suggestions for getting the most out of the debriefing:
Write a few guidelines on the easel pad before you begin:
Stick to what you observed.
Focus on the most serious problems.
Objective: a list of problems we’ll fix in the next month.
It’s your meeting, so don’t be afraid to run it. Encourage participation, but be clear that it’s not a democracy. You’re in charge, by virtue of the fact that you’re (presumably) the person in the room who knows the most about usability.
Have a laptop available with the recordings of the sessions in case you want to check anything.
You don’t have a lot of time, so keep people on track. Briefly stated opinions are fine, but don’t let it degenerate into “religious debates.” Keep bringing the discussion back to what they actually observed. “Are you saying that because of something you saw in the tests?” “Did anybody in the tests have that problem?”
Run a clean meeting. Acknowledge every contribution. No belittling allowed.
When working your way down the list and discussing fixes for the top ten problems, don’t skip any of them. The point is, since these are some of your most serious usability problems you should be doing something about all of them. You don’t need perfect and permanent fixes—and in fact you want to do as little as you can—but you need to do something.
You’ll often hear people say things like “Yes, that’s a big problem for users, but it’s going to go away next month [or next year] when we roll out Project Overlord. So there’s no point in putting effort into fixing it now.”
Always treat these claims with a certain amount of skepticism.
[ 107 ]
chapter 10
We all know that in the real world there’s always a more-than-reasonable chance that Project Overlord will end up getting delayed, scrapped, or modified beyond recognition. And even if it does come to pass, estimates of when it will happen are almost always optimistic. In the meantime, the problem exists and will continue to cause your users grief.
Rather than buy into this, ask, “What’s the smallest change we can make right now that will at least smooth over this problem for most people?”
The small, non-honkin’ report
After the debriefing, it’s a good idea to summarize this month’s testing in a short email. By short, I mean it should take no more than two minutes to read—and no more than 30 minutes to write. Think bullet points, not paragraphs. It should cover
What you tested
The list of tasks the participants did
The list of problems you’re going to fix in the next month as a result of what you observed
Let people know how they can watch the recordings or clips if they’re interested and when the next tests will be.
FAQ
Are there other ways to run the debriefing? I’ve been in sessions where there were an awful lot of Post-its being stuck up on big boards….
Yes, there are many ways to skin this particular cat. Basically, you have a lot of people’s observations and opinions, and you need to merge them into something resembling a consensus about next actions. It’s a classic business problem, so people have come up with a lot of different ways to do it.
Do whatever works in your context. Just don’t lose sight of the fact that the outcome of the meeting should be a commitment to fixing the most serious problems.
[ 108 ]
debriefing 101
Are you saying I can’t fix things like typos because they’re not the worst problems?
No, you can and hopefully will fix many other usability problems based on things you noticed during the tests. Feel free to keep your own list of problems and fix them yourself or pass them on to others who can fix them. The purpose of the debriefing, though, is to make sure your finite resources are focused on the most serious problems first.
[ 109 ]
chapter
chapter11
11
>
The least you
can do™
why doing less is often the best way
to fix things
[ 110 ]
It was in Chinatown.
What were you doing there?
Working for the District Attorney.
Doing what?
As little as possible.
—J. J. GITTES (JACK NICHOLSON) AND EVELYN
MULWRAY (FAYE DUNAWAY), IN CHINATOWN
Here’s the most important thing I’ve learned over the years about fixing the problems you discover in a usability test:
When fixing problems,
try to do the least you can do.
This means that when you’re deciding how to fix a usability problem, the question you should always be asking is
“What’s the smallest, simplest change we can make that’s likely to keep people from having the problem we observed?”
But I find that people often resist this idea. Once they get into a debriefing, they find all kinds of reasons not to do as little as possible:
“If we’re going to fix it, we want to do it right.” People seem to think that fixing a usability problem means finding a complete and permanent solution: “Eliminate the problem.” I tend to take a much more pragmatic view: “Make it better for our users right now.” To me, this is a case where the perfect is the enemy of the good.1
1 Yes, I do know that a more literal translation of Voltaire’s “Le mieux est l'ennemi du bien”
is probably “The best is the enemy of the good.” Thanks, though. It’s always nice to know people are paying attention.
[ 111 ]
chapter 11
Quick fix
“Doing it right”
A lot fewer people will experience
Almost no one will experience
the problem
the problem
Easy to implement
May require a lot of work
Probably done in a few days
May take weeks, months, or longer