what the research is about and why it is being done (why you are here);
–
their role in the research (how they can help);
–
thank them for their time;
–
make sure they know that it is the manuals and their designers who are on trial, not them;
–
the fact that you want to make notes on what they say/do (do they mind this?);
–
how you will go about the interview (you will start by asking them some general questions and then go on to detailed questions about the guide).
2
Find out what kind of experience people have with using guides; the feedback from those who are less experienced may be more important than that from practised users. Ask the following questions:
–
Do you use guides?
–
What do you use them for?
–
What do you think makes a good guide (you could ask them to name a guide that they find practical to use and probe as to why this is).
3
Explain the task and hand him/her the guide. Give him/her time to look at this and ask questions about it and about the task. Reassure them once again that they are not on trial – the manual is. Emphasize that they can ask for help at any stage of the test.
Testing the field guide 227
4
Observe the task and record your observations, including all ‘finding/searching’
activities on a form such as that in Table 9.2. What do they do first – for example:
–
Look at the book to see how it works?
–
Look in the index?
–
Look in the contents pages?
–
Look at any preliminary information?
–
Flick through the book randomly?
–
Make a closer observation of the specimen; if so, what are they looking at?
Where do they go next? Watch carefully to see what people do from first action to the end of the task.
Note any hesitation or annoyance, as well as the following actions: repeatedly going back to the same place; trying to use more than one part of the book at the same time; keeping their fingers on more than one page to keep them all open.
Note any questions asked and your answers.
5
Use semi-structured interview methods after this activity in order to probe about the use of the guide by asking the users why they used the components in that way, and reminding them to give their opinions. At the end, ask the following questions:
–
Are you sure of the identification/information that you have found?
–
Why/why not?
If the respondent did not use some of the access tools, ask why not. For example:
–
Why didn’t you use the key?
–
Did the index of common names help?
–
Do you think it is necessary to include the distribution maps?
6
Record the results – for example, on the photocopy of the mock-up. Note:
–
How long did it take the respondent to identify the plant when using the guide?
–
Was the identification correct or not? (If the interviewer does not know the plants well enough to be absolutely sure of the identity, the only way to conduct this test properly is to collect a voucher specimen of the plant used in the test and to send it to a botanist who can give you the correct scientific name.) See ‘Documentation: Instructions and forms for the test’ for some suggested ways of documenting the results and observations.
other, and it should be made clear that they are valued representatives of the target user group.
It is vital to ensure that the team conducting the tests understands the importance of adopting a respectful attitude towards the participants. The team is there to guide, provide information and help the testing process run smoothly, and in no event to assume the role of an interrogator or presume greater knowledge than the participant, even if this is, in fact, the case. An attitude of this kind may make the participant, or even the whole group, feel inhibited or under attack, and this could greatly reduce the value of the entire testing process, as well as perhaps never being able to work with those individuals in the future.
At both the beginning and the end of the workshop, thank the participants for giving up their time, and re-emphasize the importance of their taking part in the tests to find out more about the target users.
228 Plant Identification
Dynamics of the workshop
Dynamics are not only important when it comes to putting participants at their ease or introducing an element of fun into the workshop, but also when planning the methodology used in the activities and exercises themselves. The dynamics exert an important influence on the success of these, both in terms of achieving the objectives of the tests and maintaining the cohesion of the group so that the workshop goes as well as possible.
As far as the introduction of team members and participants is concerned, it is important that all individuals are given the opportunity to introduce themselves; this is a valuable moment, allowing them to be briefly in the limelight. The introduction can be done in a variety of ways – for instance, a draw is made and participants put into pairs, and each individual introduces the other to the rest of the group. To do this, they have to get to know each other a little. There are many ways of setting up an introduction; but the fundamental thing is that it does occur.
For the exercises themselves, the participants may be organized in groups or pairs, and one should always try to match individuals, both on a personal or professional basis, bearing in mind the desired result of the test. When testing a textual description of a species, for example, in a group or pair including both a botanist and someone who has never identified plants before, the botanist will be the only one to provide an answer.
This will result in an unrealistic evaluation of the text and whether or not it serves to identify a plant: the non-botanist will let the botanist answer so that the evaluation process is pointless. A way of avoiding this situation would be to organize the participants into pairs without experience of identification, and to put the botanists together in a control group so that they can assess the material and check that the information contained in the text is correct, or discuss among themselves whether any other information could be added to aid the identification process.
If at all possible, the tests should involve activities in the field. It is best to plan such activities for the morning sessions, and participants should be divided into small groups, each with a group leader and someone to record the observations, comments, questions and problems that come up during the exercise.
Team members and workshop leaders must have well-defined roles. A workshop leader or coordinator will always present the activities to the whole group, keep track of time, explain how to do the exercises and use the materials, and keep an eye on the work of the team conducting the tests. For each small group or pair, there should be an observer to accompany them during the activities, observe participants’ reactions, record the difficulties experienced and the positive aspects of the exercise, and provide guidance, if necessary.
A general meeting or feedback session may be organized, either at the end of each activity or in the form of a general debate at the conclusion of the workshop. This is an opportunity for all members of the group – participants and team – to share opinions and doubts relating to the test material. People often have very different questions and opinions, and each individual may have a very personal and specific point to make on the activities and test material. During the exercises, the observers will record these comments, which can then form the basis for debate. The debate can also be used for testing and chec
king preferences for particular guide formats, layouts, colours, fonts, textual matters, etc.
Testing the field guide 229
Evaluating the workshop
At the end of the workshop, some form of assessment on the part of the participants is a good idea, and this should include an evaluation of the venue and its facilities, the organization of the project, the type of material used, the work of the project team, the activities themselves, and any other questions for which feedback would be useful if holding another workshop in the future. The team can be involved in this process, but it is not strictly necessary. The evaluation may take the form of a general meeting at which everyone can express their opinions, answer evaluation questionnaires or just mark positive and negative points on a coloured form. During the evaluation meeting, questions can either be suggested by the team, raising precisely those points that are judged to require assessment, or the session can take a freer form, with short presentations by participants on any area that they like.
PREPARING MATERIALS FOR TESTING
Each test has its own objective, and both the test material itself and the material which will serve as the methodological basis for the test must be prepared accordingly.
Preparing printed pages
If you are testing illustrations, the quality of the paper, ink and printing process is extremely important. Fuzzy photos do not help anyone to decide what illustrations to include, unless you plan for the final version to also have fuzzy photos! This can be expensive, so think about numbers needed. A set of photos will be required for each plant, including images of those features to be tested (a flower in its entirety; habit (e.g.
whether tree, shrub or herb); a flower enlarged; a leaf; a branch in flower; etc.).
Producing a set of illustrations for every participant in the test is very expensive, and it is useful to set up a system where each person present can analyse a set at a time, and each set can be passed around all participants, in turn. It is also very costly to print all of the pages of a guide to a publishable quality (including good-quality paper, colour photographs, drawings, etc.), although this is important if the material is to be tested in the form in which it will be published. To save resources, one optimized version can be produced, and the others done to a lower-quality specification. The most important point is that all participants have access to material that depicts the quality of the published guide so that they can express their opinions about it.
With regard to checking texts with descriptions of the plants, it is best, if possible, to test all pages of the guide relating to the plants included and to prepare them in a format as similar as possible to the eventual published guide. This is often hard because of time pressures or the lack of resources for printing the pages; but even so, some pages can be tested in this way.
The quality of printing and the type of paper used is a relevant factor in the test material, and if there are different options regarding this, the materials (whether individual pages or the whole guide) can be prepared accordingly. For example, the paper type could be tested by producing samples on matt black and gloss couche paper, and on recycled paper, in white or beige. This would allow users to express their views on the most appropriate options.
230 Plant Identification
When preparing the material, it is also important to try to test different groups of plants or animals so that the participants do not become bored. Another reason to do this is that once the plant has been identified, the participants may, in the next exercise, choose it again on the basis of familiarity, and not because they identified it using the data provided. When a test is carried out using photographs of a plant, followed by another with drawings of the same plant, the participants simply choose the right species because they already know the plant, not because the drawing helped them to identify it.
In this case, the test is a waste of time since it is impossible to conclude whether the drawing was useful in the identification process.
Preparing specimens
For testing characters, keys or usability, the testers will need to examine real plants, either whole in the field, or cut specimens brought to the workshop. If specimens are cut, they must obviously be the freshest possible – it is no good asking people to identify specimens that were collected the day before and have since wilted. Above all, the specimens must actually include the features that are necessary for identification. If your field guide relies strongly on bark characters and you cannot bring a sample of the bark to the testing workshop, you must go to the field to test your guide.
Preparing a mock-up
Probably the commonest misjudgement made by field guide writers is to produce a final draft only during the last days of the project. Be safe and aim to produce a working draft
– perhaps missing the odd item, but otherwise complete with respect to number of pages, illustrations and indexes – at the latest 75 per cent of the way through the project life span, leaving the last quarter of the project to correct problems that are apparent only when everything is in place.
The main issues for those producing the mock-up are:
•
The quality of the reproduction. A general rule of thumb is that the illustrations should be as close as possible in size and colour to the final product.
•
The extent to which all text, illustrations and access systems need to be included to make the usability test useful. You may be able to leave out some information or some species, as long as the overall structure of the guide is there and none of your tests involves the missing information.
•
Who should be involved. The team who have been involved in producing and pre-testing individual aspects of the text, illustrations and access systems need to be involved in producing the mock-up since they are the ones who know what the final product should contain, what it should look like and what its purpose is. However, there are advantages to bringing into the team those individuals (graphic designers, printers, etc.) who will be responsible for the final production of the guide (see Case studies 9.1 and 9.2).
These decisions will depend upon the purpose for which the guide is intended and the amount of time, money, skill and equipment available for producing the mock-up and carrying out the usability test.
Testing the field guide 231
CASE STUDY 9.1 USING A GRAPHIC DESIGNER TO PREPARE
THE MOCK-UP IN BRAZIL
The production of the guide to forage legumes of the Caatinga (Costa et al, 2002, a spiral-bound colour photo guide to 21 species that are useful to farmers in a semi-arid region of northeast Brazil) turned into a major participatory exercise. The authors found themselves very occupied with the consultations and workshops in order to ensure that the guide was suitable for the needs of the farmers and extension workers in the area, and felt that they lacked the time and specialist skills to prepare the page formats. As the testing progressed, they decided to contract a graphic designer to produce the pages of the mock-up to a similar quality as the published guide in order for the tests to provide the most reliable results possible.
Although she did not have experience in producing field guides, the designer was able to work to create the material in a participatory way, listening to the opinions of the team, so that the end product reflected all decisions taken in the workshops and met the needs of the target audience.
The designer was then given all the text, photos and drawings, as well as the guidelines on how the material should be presented according to decisions taken with the target audience in mind. Constant surveillance of the production process was needed to ensure that it met the testing requirements.
Once engaged, the designer spent 30 hours producing the material. In addition, the team had plenty to do: to produce simple pages with printing and photocopying, they needed 12 hours to scan the 105 photographs and illustrations, 5 hours to format the 60
pages of text and set it on the page, 20 hours to set up and edit the test pages
, and 3 more hours to print a high-quality original on the laser printer. Five more hours were required for laser photocopying on coated paper. The printing costs for the original involved approximately two black-and-white and two colour cartridges, as well as the cost of the paper.
High-quality (laser) photocopies were then made so that the guide could be tested simulta-neously with various user groups.
This professional approach came at a cost. The designer cost US$615, producing a layout for the mock-up and revising it for final publication. In addition, to ensure her interaction with the stakeholders, she spent time in the workshops at a cost of US$92 per day, not including board and accommodation. And, finally, the cost of printing, reproduction and binding the mock-ups was US$184.
As the pages were checked during testing, many questions arose regarding form, colours and graphic design possibilities. We realized that the presence of the designer during testing could make the tests more dynamic and more useful for improving visual aspects of the guide. The designer could be present to resolve doubts and suggest specific solutions, keeping in mind the costs of the different options.
The designer also participated during the testing of the page proofs (a prototype for the guide), when plants were identified by using the guide, in order to give her a full grasp of the users’ requirements so that the visual presentation would meet the needs of the target audience. This also enabled her to provide technical advice and involvement in discussions on what format to choose for the guide, possible fonts, colours, artwork, the paper to be used and its availability and, of course, the costs of the various options.
Although this appeared to be a luxury, involving the graphic designer during the participatory preparation of the guide made the process run more smoothly, and the end product was published on schedule.
Source: Maria Theresa Stradmann, Ana Paula Ferreira, Teonildes Nunes and Jorge Costa
232 Plant Identification
CASE STUDY 9.2 PREPARING THE MOCK-UP IN THE OFFICE: THE
PROS AND CONS OF A MORE ECONOMIC APPROACH IN BOLIVIA In contrast to Case study 9.1, the editors responsible for producing the mock-up for the guide to the useful plants of Bajo Paraguá (Vargas and Jordán, 2003) decided to carry out all of these tasks in the office. They had access to high-quality printers and an interest in visual presentation; so although it was time consuming, they saw this as a way of avoiding high costs at the testing stage and of developing their own skills.
Plant Identification Page 38