9
Testing the field guide
Anna Lawrence, Patricia Norrish, Maria Theresa
Stradmann, Israel Vargas, Edwin Magariños, Jorge Costa, Claudia Jordán and Teonildes Nunes
INTRODUCTION
Throughout the process of writing a field guide, decisions on, for example, content, the type of illustrations and keys will have been taken with representatives of the end users and balanced against the resources available. Such activities reduce the possibility that a guide will be unacceptable and incomprehensible. However, the ultimate test of the guide is not only that it contains useful information, presented in a format that people like and understand, but that it is usable for the kinds of tasks which people want to perform. Chapters 5 to 8 have all pointed out the need to both pre-select formats in consultation with users, and to test the chosen formats at an early stage to ensure that they are understood by the users.
Both the format and content of each component, as well as a full mock-up of the guide, must be tested in realistic conditions by the kinds of people who will use the guide. The mock-up should also be as realistic as possible without going to the expense of final printing. This stage of testing is often called usability or performance testing.
Because the basic process of testing is the same for components and for the whole guide, we describe a model (outline) of the process first, and then give specific examples of how to elaborate upon this for the different aspects of the guide that need to be tested.
This chapter discusses the need for usability testing; the ways in which a reasonable mock-up can be prepared; what needs to be tested; ways of carrying out the testing; and how to analyse and use the results to make adjustments to the guide.
Reasons for testing components
As explained in Chapters 3 and 7, users of the guide do not necessarily have the same knowledge, perceptions, worldview and communication style as the people who produce the guides. It is often surprising to authors to find out not only that their language is impenetrably technical, but also that people from different cultures or educa-
216 Plant Identification
tional backgrounds may be put off by certain fonts, letter size or density of uninterrupted text, or, perhaps even more surprising, that different people interpret pictures in different ways. In tests conducted in northeast Brazil, highly accurate ink drawings of medicinal plants had been prepared carefully, with parts enlarged inside circles, as if under a microscope, to show key identification characters such as hairs or glands. The agricultural technicians testing the guide did not recognize the enlarged parts within circles as magnifications, but instead perceived the illustrated plants as diseased, having extraordinary distortions of certain parts.
In another example from Bahia, Brazil, a workshop with farmers was used to test the usefulness of photographs for plant identification. A selection of photographs was chosen for each species, and among these was a photo of the flower on its own, enlarged if the flowers were small. For the botanist, it was clear that this picture was important, and that it would often help them to identify the plant. The farmers, however, who are not used to seeing the flower in isolation, sometimes did not recognize it in this type of photograph. The photos that helped them to identify the plants were those which depicted the branch in flower or fruiting, illustrating the plant in its natural environment. Without this test, resources would perhaps have been wasted on producing images that would not be useful for the target users, and the guide would have failed in its objective. Even more importantly, the farmers felt deeply involved in the production process and enthusiastic about the product.
Reasons for testing usability
As explained in Chapter 2, ‘usability’ refers to the properties of the guide that enable users to achieve their objectives successfully – in other words, to identify plants, find information about those plants or to recognize that the plant in question is not in that particular guide.
Usability testing of a mock-up of the guide is about seeing how people find their way around the guide, and whether there are any serious obstacles to using it that need to be addressed before final printing. It can pick up issues that may not have been noticed earlier during pre-testing of individual components of the guide, such as portability, durability, binding defects and, most importantly, identifying the right species among the whole mass of species in the guide.
This final evaluation of the field guide is obviously useful for authors to help improve the accuracy and usability of the guide; but it is also an important stage for users who have participated in the production to reinforce their feeling of involvement.
One user made the following comment: ‘Seeing our names in the book alongside the others has given us an incentive to keep helping with the production of the book.’
OVERVIEW OF THE TESTING PROCESS
It is difficult to test several components of a field guide all at once. For example, if the aim is to test whether or not the diagnostic characters of the species described in the text are important and helpful for the user to identify the plants, the materials used in the test must consist only of the text with the information you want to check, as well as samples of the plants that are the object of identification. It is important that neither
Testing the field guide 217
photographs nor drawings should be used since these will make it impossible to ascertain whether it is the clarity of the text or the characters described that enable identification.
The first step, therefore, is to define exactly what it is that you are testing. If illustrations, are you testing which illustrations people prefer, or which ones help them to reach an accurate identification? What are the possible options? They may include preferences already stated by users at the consultation stage, as well as innovative ideas for illustration that the authors have seen elsewhere. How can you gather or prepare material to test these options? And how many species would you want to include? For example, if you want to compare the usefulness of ink drawings with photographs for identifying trees, you might prepare illustrations of both kinds for a sample range of species that covered all the difficult aspects of identification – such as leaves, bark thorns or fruits.
Table 9.1 sets out some possibilities for testing procedures. Each test must have an underlying question, such as ‘Which format leads to the most accurate identification?’, and materials and method can be chosen accordingly to match the question. But, of course, there are many more detailed questions to be asked, such as ‘What is it that you like/dislike/understand/don’t understand about this picture/description/key, etc.?’
There are also decisions to be made about how the testers will get on with their task.
Sometimes the testers (who represent potential users of the guide) are left to examine the material or to carry out a task, and then the facilitators can interview them to ask these questions. At other times, especially if the testers are not used to following written instructions or may become nervous about doing so, the facilitator can guide the tester through the steps. However, with usability testing the testers must be left to find their way through the guide without assistance since this is the point of the test.
In all of this, the role of observer is extremely important. Often, the testers will find it hard to explain how they used the guide, or will not notice factors such as their speed in identifying something, puzzled expressions on their faces or repeated looks at one particular illustration. All of these are clues to the facilitators about how easy or difficult it is to use the guide.
After these activities, it is important to round off with a discussion to summarize the testers’ experiences and probe into their reasons for liking or not understanding something. If possible, this should take the form of a group discussion so that opinions can be shared and consolidated.
In summary, then, the whole process involves the following:
•
Plan: what will you test, and what needs to be recorded and o
bserved?
•
Invite: who will test this? Invite them, explaining why you want them to test the field guide and what the arrangements are.
•
Gather or prepare materials: specimens must be freshly gathered and display the characters of interest; plants in the wild must be easily found and clearly marked; illustrations, descriptions or the mock-up must obviously be ready in time for the testing workshop.
•
Prepare instructions for the test itself: these should be read by the testers or be used by the facilitators.
•
Prepare forms for the results to be recorded: these are filled in by the testers and facilitators (who must record their observations).
•
Organize logistics: plan travel, food, space and workshop materials.
218 Plant Identification
•
Run the tests, usually in the form of a workshop.
•
Record results, observe behaviour, discuss and explain experience.
•
Reflect on outcomes after the workshop.
These steps are described in more detail in the following sections.
Timing of testing
As indicated in Figure 2.1 (page 16), testing occurs at various stages of field guide preparation. Some components of the guide need to be tested as early as possible in the production process in order to avoid wasting time and money (for example, in producing illustrations that are not of the required type) or in order to guide the information-gathering process (for example, by finding out which diagnostic characters your guide users find more practicable to use in the field). Other components cannot be tested until you have enough information to prepare the content. This applies particularly to a thorough reading of the descriptions of all species to check for any incorrect or contentious information and, of course, to the final testing of the full mock-up, which must contain as many of the final descriptions and illustrations as possible, in the right order, and with keys and indexes where required to help the user find the correct species.
METHODOLOGY
An important issue to consider when planning and choosing a suitable methodology is selecting an approach that is appropriate for the readership. This necessitates a clear definition of the target users of the field guide as the participants in the tests must represent a sample of this group. When testing material for farmers, for instance, given that the material to be tested (content of the guide) contains a specific type of language, the methodological approach used must also be in a language understood by this group and in accordance with their education. For example, if the users include people who cannot read or write, there is no point in creating a questionnaire for them to read and answer.
It will be necessary to think up ways for them to analyse the material, perhaps involving helpers to aid them with the tasks, although it is, of course, important that the participants should not feel embarrassed. Equally, if the target users are eco-tourists, there is no point in preparing long tests or questionnaires with a large number of questions as these users, on holiday, are focused on enjoying themselves and are not going to spend a long time answering a boring questionnaire.
Each test explores the ability of the guide to answer a particular question (for example, ‘What species am I looking at?’). The basic aim is to define a method that challenges that particular question, to carry out this defined task using the materials available, to record the process followed by the testers and the conclusions that they reach, and to analyse the reasons for them following that process and reaching those conclusions. In short: present task; ask testers to do task; record and observe; analyse through interview and reflection. Table 9.1 shows how this basic core method is used in a range of tests for illustrations, text, keys and the whole guide.
Thought must be given to how you will make the testers feel comfortable and how to explain the purpose and procedure. In order for the tests to have validity, there needs to be uniformity on this, and what is going to be said should be set out in some way,
Testing the field guide 219
either as a statement to be read or as a set of guidelines on what must be covered. It may be important to relax and reassure people that it is the guide (not themselves) that is being tested, and that inability to use the guide is nothing to be ashamed of, but rather a very useful result. Many people can lose their concentration if they think they are being observed or feel under pressure. For this reason, it is sometimes better to conduct evaluations where people are left alone to get on with the guide use at their own pace. If necessary, usability can be assessed by asking users questions at the end once they have been shown which species they identified correctly and which they did not, discussing at this later stage where the guide was misleading.
The skills needed by the facilitators are:
•
structuring a task based on the guide:
–
carrying out structured observation of how people address the task to find out if and where the process breaks down;
–
recording of the structured observation;
•
preparing a semi-structured interview:
–
carrying out a semi-structured interview with the potential user (before, during and after the task as necessary) to determine existing levels of skill in using a guide and the specific problems/successes with the guide being tested.
Semi-structured interviewing is described in Chapters 3 and 7, and the section on
‘Questionnaires’ in Chapter 7 will also be helpful. Thought needs to be given to the questions you will ask at the start in order to determine whether people have experience of working with guides or not (their experience will affect the speed and ease with which they carry out the task), and questions that need to be asked at the end of the task to determine why people did what they did.
Setting a task that tests the aspect of the guide you are interested in is a logical process. Make sure that all aspects of the component are tested. For example, if you are asking testers to name the species illustrated, ensure that a range of different kinds of species are included, and that different parts are illustrated or different types of illustration are used. Sort these randomly so that one illustration does not help the tester to recognize the next. Some examples of tasks are given in Table 9.1.
In addition, the facilitators need to be particularly skilled at detached observation, and at recording the behaviour and discussion that occurs during the test. Observation and interpretation of a situation, informed by an understanding of the situation from the point of view of the participants rather than the observer, is a particular challenge if you have been closely involved in the development of the guide and have strong opinions about how it ‘should’ be. But the tests will not be objective if you let your concerns get in the way of understanding the blocks and responses that the testers have to the guide.
These observation skills are particularly useful in the usability test, which is a procedure specially designed for testing the use of the full mock-up.
The usability test
The aim of the usability test is to see the guide being used to solve a set task, in as realistic a situation as possible. This will preferably be in the field. If it is difficult to conduct the usability test in the field, a simulation can be organized by bringing the specimens to the
220 Plant Identification
-
-
, or to
in
sorted
.
.
acilitators observe testers for signs of
acilitators observe testers for signs of
acilitators observe testers for signs of
Method
Testers work individually or
groups to identify randomly
Depending upon the testers, facilitators
can ask them to report verbally what
&n
bsp; species they think it is and why
write their answers on a form.
F
confidence or worry
Interview testers afterwards to under
stand their reasons for successful or
unsuccessful identification.
Individual testers read texts.
F
confidence or worry
Testers and facilitators discuss texts in
groups to reach consensus and also to
seek alternative words for those which
are not understood.
Testers work individually or in groups to
examine specimens or pairs of
specimens.
Depending upon the testers,
facilitators can ask them to report
verbally what characters they observe
or differences they find between the
species, or to write their answers on
a form.
F
confidence, worry or confusion.
Interview testers afterwards to under
stand their reasons for successful or
unsuccessful completion of the task.
,
of pictures.
(flower
types
parts
Materials needed
Range of illustrations for
each species. These may
be different
illustration (drawing,
photograph) or illustrations
of different
fruit, whole tree, branch,
bark) depending upon
what you are testing.
The same species
description formatted in
different ways, or the same
species, described using
different words.
Specimens for comparison
or marked
species in the field.
If you are asking the testers
how they recognize the
species, the specimens
must be in the field so that
all possible characters are
available, and must be
Plant Identification Page 36