How to Design and Report Experiments
Page 39
Hopefully, you can see from this example what you should be trying to achieve in the main part of the discussion. You are trying to see how your results fit in with previous work in that area, and you are trying to use your findings to clarify unresolved theoretical issues. We tried to do three things in this section of the Discussion of our paper. The first was to demonstrate a novel phenomenon: distorted faces remain highly recognizable. This is interesting, but it’s just an isolated ‘fact’ as it stands. Therefore our second aim was to show how this fits in with what’s already known in this area: other manipulations, such as inversion, have been found to disrupt recognition considerably. By discussing our findings in relation to previous work, we start to build up a bigger picture of what’s going on when we look at a face: we’ve found a particular manipulation that doesn’t have much effect on recognition, other people have found manipulations that do, and so we can use these snippets of information to start to develop an understanding of face recognition in general. Our third aim was to show how our findings could be used to assess the merits of various theories of face recognition; while one study is probably unlikely to settle an issue like this, nevertheless our results support some theories and fail to support others. (Note that, in science, we can’t prove a theory, but we can provide support for it, and we can certainly disprove a theory by producing findings that it cannot explain. In this particular example, we are able to conclude fairly conclusively that face recognition cannot be based on simple measurements taken straight from a face – because if it were, recognition of distorted faces would be very difficult. We have therefore disproved any theory of face recognition that’s based on simple-minded measurements of facial proportions.)
Doing a jigsaw is a good analogy for all of this. Imagine that there are hundreds of people who are all working on a jigsaw. Nobody knows what picture is depicted on the jigsaw. Each of us brings an individual piece to the jigsaw every now and again, and tries to fit it in with the other pieces that are lying around. As we are trying to fit pieces together, we come up with theories about what the picture is of. Sometimes – but not often – a whole load of pieces fit together, and you suddenly realize that you’ve got a fair bit of the jigsaw done. On other occasions, you try to fit the pieces together, but they don’t all fit correctly, or you see that a crucial piece is missing, or that you’ve forced together pieces that don’t really belong together. This is essentially what you are trying to achieve in this part of the Discussion: you are trying to tell everyone (i.e., other scientists) what new parts of the jigsaw you have found; explain how you think they fit in with the other bits that are already on the table; and discuss a few of the theories about what’s in the picture on the jigsaw, arguing in favour of some possible interpretations and against others.
14.3 Discuss the Limitations of Your Study
* * *
You should show that you are aware of the limitations of your study, and the conclusions that can be drawn from it. These may include possible methodological problems: maybe next time you’ll use a lower shock voltage, so that more participants survive to contribute to the results; or perhaps the memory scores of the Attention-Deficit Hyperactivity Disorder children might have been higher had your rewards for learning a list of items not been sweets containing food-colourings. Another possibility is that there are limitations on the generalizability of the results obtained: for example, girl guides are not necessarily representative of the general population.
Don’t go into an orgy of self-abuse: no study is perfect, and just because you left one of the participants with their head in the condensed soup for six seconds longer than the rest, it doesn’t necessarily mean that your results are wholly invalid. Also, avoid mechanically trotting out standard problems with experiments. Some students will routinely write statements such as ‘it would have been better to use more participants’ or ‘the experiment should be done again, using a more representative sample of the general population than undergraduates’. Don’t include statements like these unthinkingly. At one level, the statement about participant numbers is so obvious that it is not worth saying, because it would be true for any research that was done. (No matter how many participants are used, it would always be better to run more, to get a more reliable sample). Perhaps what you really mean is that, with hindsight, you don’t think that you ran enough participants to find any effects of your experimental manipulations. If that’s the case, and it would have been better to use more participants, the obvious question is – why didn’t you do that in the first place? The issue of how many participants to use should have been considered before the study was conducted anyway (see the section on statistical ‘power’ in Chapter 4).
The second statement, about students being unrepresentative of the world at large, is not necessarily true. It all depends on the nature of the phenomenon under investigation. If the study is influenced by factors such as educational level, IQ, conservatism, youth, etc., then it is true that using students may produce a biased picture. For example, it’s probably highly likely that, overall, students’ attitudes to race relations are different to those of the general population. However, if the experiment is on something which is unlikely to be affected by student characteristics, such as low-level visual perception or memory span, then students are as ‘representative’ of humanity as anyone else.
Essentially, this part of the Discussion is an opportunity for you to demonstrate your ability to be self-critical and sufficiently evaluative about the research that you have performed. The idea is that you should anticipate the reader’s criticisms of your research and preempt them. ‘Attack is the best form of defence’, in research as it is in life. In the case of a journal article, by the time the reviewer has reached this point in the report, they have thought of various problems and unresolved issues that have been raised by the research; this section is the researcher’s opportunity to try to put the reviewer’s mind at rest, effectively saying ‘yes, with hindsight I’m well aware that using a repeated-measures design was not the best way to tackle this problem, and that the results for group X are rather odd; however, I can still persuade you that these problems haven’t totally invalidated my study, and that my results are still worth publishing’. If you are writing a lab-report, you have a comparable task – to persuade your tutor that you have a sophisticated understanding of experimental design, by showing that you are aware of the methodological and conceptual weaknesses of what you have done.
14.4 Make Suggestions for Further Research
* * *
Finish off the discussion with some suggestions about what issues remain unresolved, and where the research could go from here. You could make some brief suggestions for future studies. However, be intelligent about this, and make sure there is some rationale for these suggestions. Don’t just write something like ‘Future studies might look at gender differences on these measures’ or ‘It would be worth repeating this experiment using a group of Botswanan participants instead’. Explain briefly why this would be worth doing, and what it might show.
14.5 Draw Some Conclusions
* * *
Consider rounding off the Discussion with some final conclusions about what your study has shown. These should be at the end of the Discussion section itself, rather than in a separate section labelled ‘Conclusion’. Try to avoid merely repeating what you said earlier in the Discussion. Avoid hackneyed, empty phrases like ‘Clearly, much more work needs to be done in this fascinating area’. Statements like this ultimately say nothing other than that you are desperately trying to ‘round off’ your report. (It’s self-evident that more work could be done in any area of psychology, and we can safely assume that it’s fascinating to someone!)
14.6 Summary
* * *
The purpose of the Discussion is to interpret the results that you have found, in relation to previous research: how do your findings fit in with what’s already known on the topic in question?
The Discussion h
as three main parts: a brief recapitulation of your main findings, a discussion of how these results fit in with previous research in this area; and a discussion of possible problems and limitations with your study, perhaps with intelligent suggestions for future research.
As with the Introduction, make sure that the research and theories that you describe are relevant to your own study.
15 Title, Abstract, References and Formatting
* * *
The title, abstract (summary) and references are the last bits of the report that you should write, but they are important nevertheless: the title and abstract provide the reader with their first indications of what your study was all about, and the references provide them with the opportunity to find out more by reading research that is related to yours. For an assessed report, there’s also a pragmatic consideration: the title and abstract will give the marker a first impression of the likely quality of the rest of your report, and the reference section will give them an idea of how well read you are, and how thorough and careful you have been in producing the report. So, for a variety of reasons, although these may be small sections, it’s important to get them right.
15.1 The Title
* * *
This should be clear and specific, but not too detailed. For example, suppose we had done an experiment in which we had investigated whether or not having breakfast affected people’s ability to concentrate later in the day. A good title would be something like ‘The effects of breakfast on mid-morning concentration levels’. Avoid using a title which is vague or not detailed enough, such as ‘An experiment on concentration’ or ‘Breakfast and concentration’. Similarly, avoid producing a title which is too long and convoluted, and unnecessarily detailed, such as ‘The effects of having breakfast or not having breakfast on the concentration levels of my fellow students, as measured with a selective attention task’. The American Psychological Association suggest the title should be about 10 or 12 words long.
Suppose we were interested in gender differences in the emotions that Navaho Indians display in relation to tear-jerking stimuli (such as Walt Disney’s ‘Bambi’ film). The title should give a clue to the important aspects of the study. Thus ‘Emotion and Indians’ or ‘Navaho emotional expression’ would be uninformative titles: ‘Gender differences in emotional expression in Navaho Indians’ would be preferable, as it encapsulates, succinctly, the dimensions of interest. Anyone interested in Navahos, gender differences or emotional expression (or maybe all three!) would be able to spot at a glance that this was a study that they might well be interested in knowing about.
Box 15.1: Examples of good and bad report titles:
‘The ‘Beer-Goggles’ effect’ Too vague.
‘Investigation into the effects of drinking 6 pints of Guinness in a night-club in Brighton at 11.30, while listening to drum and bass played at 100 db through a good sound system, on 12 men’s and 15 women’s ability to accurately judge the attractiveness of members of the opposite sex’ Over-specific, over-detailed and over-long.’
‘The ‘Beer-Goggles’ effect: the effects of alcohol on ratings of the attractiveness of members of the opposite sex’ OK (but still a bit long).
A good rule of thumb is to make sure that your title includes a mention of the independent variables in your study (i.e. the factors that you, as experimenter, are manipulating in the study), and the dependent variables (the things that you are measuring). (See Chapter 2 for more explanation of independent and dependent variables). Thus, in our title ‘Gender differences in emotional expression in Navaho Indians’, ‘gender differences in Navaho Indians’ is our independent variable, and ‘emotional expression’ is our dependent variable.
Look at your title and ask yourself: does it give the reader a good idea of what the experiment’s about, while remaining fairly snappy? In the case of journals, a researcher would skim the contents page, looking for any articles that looked likely to be of interest to them: succinct but informative titles are a great aid to doing this quickly and efficiently.
15.2 The Abstract
* * *
This is the first section of the lab-report. It is placed immediately after the title, but write it last of all – it’s much easier to write it once you have done the rest of the report. Aim for a 150 word summary of what you did, what you found, and what it means. (The American Psychological Association enforce a 120 word limit very strictly). You don’t need to go into much detail – if the reader wanted the details, they’d go ahead and read the report itself. In the case of a journal article, the idea is to give an overview that will enable the reader to quickly decide whether or not the paper is of interest to them. Aim for one or two sentences describing each of the following: the reason for the experiment; the methods used; the main results found, and the principal conclusions to be drawn from them.
Don’t include references, excessive amounts of procedural detail, or details of statistical tests. Keep it simple, clear and to the point, and make sure that it’s understandable without reference to the main text.
15.3 References
* * *
In a lab-report, there are two main ways of referring to previous work. Within the text of the report itself (i.e. principally in the Introduction and Discussion), references are cited by giving the author’s surname and the year of publication. At the end of the report, there is a section entitled ‘References’, and this gives full details of all of the references for which you gave surname and date in the text. This is the so-called ‘Harvard’ system of referencing. There is another method, the ‘Chicago’ system, that is used in ‘Science’ and many medical journals. Little numbers in the text refer to the full references in the Reference section at the end of the article. In the Reference section the references are listed in order of appearance in the text. Unless you are going to submit your lab-report for publication in ‘Science’, don’t use this system.
The whole purpose of the reference section is to enable the reader to gain access to the works that you have cited throughout your report. To ensure that sufficient information is provided in a reference list for this to be possible, the American Psychological Association and the British Psychological Society have laid down very clear, specific and detailed conventions for how to produce reference lists. In what follows, we will follow the APA’s guidelines on referencing.
Conventions for References Cited in the Body of the Report
Citing works by a single author
Give the author’s surname and the year of publication, as in the following examples. You can write them like this:
Box 15.2: Examples of good and bad abstracts:
Gender differences in essay writing were examined, Essays written by ten boys and ten girls (mean age 12 years), and ten men and ten women (mean age 20 years) were assessed by 40 naive raters on four dimensions: readability, coherence of argument, degree of balance in the argument, and degree of scholarship, Raters judged the women’s essays to show significantly greater readability and coherence of argument than the men’s, There were no significant differences for balance or scholarship. No gender differences of any kind were found for the children’s essays. It is concluded that post-adolescent educational experiences may influence writing style in different ways for men and women. A good abstract. In 108 words, this specifies the two independent variables (gender and age); gives brief details of what was done, and the main results; and ends with a brief conclusion.
Previous research, for example by Riterskramp et al. (1948), has suggested that there might be gender differences in some aspects of essay writing but not others, so an experiment was conducted in order to investigate this, and see whether or not there were gender differences in the ability to write essays. First, essays were obtained from ten boys and ten girls (mean age 12 years) at a local secondary school, and from ten men and ten women (mean age 20 years) who were undergraduates at the University of Nohope. These essays were rated by 40 naive participants on four dimensions: how readabl
e the essays were, how coherent their arguments were, the extent to which the writer showed a degree of balance in the argument, and the extent to which the essay showed scholarship (in the sense of being well-researched and well-referenced). Nine-point Likert Scales were used by raters to rate each of these dimensions: in each case, these ranged from 1 = ‘atrocious’ to 9 = ‘excellent’. The naive raters judged the women’s essays to show significantly greater readability and coherence of argument than the men’s (Independent t-test for readability: t = 2.04, with 39 degrees of freedom; p < .05. Independent t-test for coherence of argument: t = 2.62, with 39 degrees of freedom; p < .05). A poor abstract: this is 390 words long (and believe me, we’ve seen students produce longer ones in real life!). It’s fairly clearly written, but the style is too discursive – it could be much more succinct without any loss of clarity. It contains too much information about related research (summarize your own study, not other people’s!) and goes into too much detail about the procedures used and the results that were found.
There were no significant differences for balance or scholarship (Independent t-test for balance: t = 0.56, with 39 degrees of freedom: p > .10. Independent t-test for scholarship: t = 1.03, with 39 degrees of freedom, p > .10). In the case of the children’s essays, there were no apparent gender differences of any kind (Independent t-test for readability: t = 0.99, with 39 degrees of freedom; p >