There is a simple test that I use to determine whether a potential graduate student is in a suitable frame of mind to undertake post-graduate research in science, and this test seems applicable to most disciplines (maybe except Maths). Ask the student how they would ‘test’ the hypothesis that ALL swans are white. Here in the UK, we have three native species of swan. They are all normally white in adult-phase, and any UK/EU student will be familiar with notion that swans are white. They think they already know the most likely answer before gathering any data, a common problem in research. So how do students go about testing such a hypothesis? There are two (or three) distinct approaches:
The first student would go to a local country park, where they would ‘look for’ white swans. They would find hundreds of white swans. They would go around all of the local parks and find even more white swans. Their preconceived notion of swans being white is supported. They come back to say “I have looked in all of the likely places, looking for white swans, and I counted several hundred. All swans are white, hypothesis proven”.
The second student would also go around the local parks and country parks, taking careful note of the number of white swans, but looking specifically for black swans, or swans of any colour other than white. They return to say “I have looked in all likely places looking for swans, and all of the swans I found were white. If there was a black swan, I think I would have spotted it, but I didn’t see any. Therefore, the hypothesis is supported (for now) that all swans are white”
The third student would go further afield, researching whether there have been any sightings of swans that are not white, and following up on such sightings, and gathering evidence that would disprove the central hypothesis. They return to say “I have seen evidence of thousands of white swans, of three different species (as determined by size, and by bill colour), which initially seemed to support the hypothesis, but I’ve got evidence of a black swan, and it is of a different species. I also have evidence of reports of a few pairs breeding elsewhere in the UK. Apparently they are escapees from exotic bird collections, and that swans are black in Australia, and other species and black and white. The central hypothesis has been disproved. I will now refine my hypothesis and undertake further research on…”
The first student knew what he was looking for and looked for data that ‘fitted’ his hypothesis, or belief, that all swans are white, and he gathered as much evidence as possible to convince himself (and his supervisor) that this is the case. I see this confirmation bias first hand all too often in science, and other sectors also (see Thomas Gilovich‘s work for some excellent studies). The second student went out looking for black or coloured swans, so was seeking to disprove the hypothesis. OK, so he didn’t quite try hard enough, but came to a logical conclusion. The third student also went out looking for coloured swans, tried that bit harder to seek out data that would disprove his hypothesis. If even he couldn’t disprove it, despite his best efforts, then he could be content with his data being of the highest quality. Reported sightings wouldn’t be enough, he needed to see one for himself. Having disproved the hypothesis, he could now focus on other specific hypotheses armed with this new knowledge e.g. whether swans are mostly white, or whether the ratio of white:black is changing over time are very different questions/hypotheses that can now tested by enquiry to further knowledge in this area. Formulating the initial hypothesis is important in designing a study, and its limitations, and data from both students 1 and 2 would correctly support the hypothesis that MOST swans in the UK are white, but not that ALL swans are white.
Now, who would you take on for a research project to test whether a proposed intervention works, given that if the intervention works, the research team will probably get a big grant to expand the study, lots of kudos and you will probably get a promotion? That depends on you aims. If it is to get promotion and you really don’t care whether the intervention works, or alternatively you are of the mind-set that you hypothesis can’t possibly be wrong for whatever reason, take on student 1 every time. You will be happy with the results! If it is to further knowledge, take on student 3, and if he is not available, settle for student 2.
None of this is new. It is a standard scenario (or variation thereof) used by many scientists to illustrate reliable methods of gathering evidence, and how easily pre-conceived ideas can bias the data generated. The biggest fear is how this mind-set of the scientist can affect the data. Might student 1 see a black swan by chance, but either a) convince himself that it was a large, dark goose, or even worse b) suppress the data for fear of not supporting the hypothesis that the professor/group leader supports? Yes, it happens a lot. Read up on the File Drawer problem, where negative studies i.e. those with effect of treatment/intervention vs control, tend not to get published, whereas studies with a clear effect do get published. If the data ‘fits’ current convention, then it’s even more likely to be published in a higher impact journal. This has a serious impact on subsequent meta-analyses. Read Ben Goldacre’s Bad Pharma. Read up on how studies are being retracted in science for both honest mistakes and dishonest practise. One potential solution is to design studies with individuals who have generated data that opposes your own hypothesis. Such Sceptic-Proponent collaborations should reduce confirmation bias, and maybe bring opposing factions together. At this point I could go on to discuss in detail Karl Popper‘s critical rationalism, and discuss falsificationism, but I won’t.
In future blogs, I will outline some of the black swans that I have found over the years. Some of them have made me unpopular, albeit temporarily in most cases, but all of them have advanced knowledge in some way, and allowed me to focus on something potentially more worthwhile. None of the black swans that I identified have accelerated my career as much as if I had generated data supporting the initial hypotheses, but I have at least published the findings in peer-reviewed journals to outline the problems to other scientists. This is a major part of scientific enquiry. It is not such a bad thing to have formulated a hypothesis that is subsequently disproved, as long as at the time, it was done with the best possible enquiry methods. It is a major part of science that we have to accept.
PS The smaller black birds in the image are coots.