The recent article in Nature on bias in research got me thinking again about an old chestnut. Publication bias. It’s everywhere. This is a particular problem with negative results, where treatment & control groups don’t show statistically significant differences. People don’t publish them as often and when they do they tend to get in lower impact journals. This is widely known as the file draw problem.
Why is this important? Well, put simply without negative results we are only getting part of the picture of what is going on. This is a problem for all branches of ecology, but particularly for field based work. For example, finding that management practice x did not significantly alter populations of species y when compared to controls may not seem that exciting. However, if it’s not published and someone else investigates the same management elsewhere and it turns out to increase the population of species y and they go on to publish this, there is a bias in the literature. This can give us a completely skewed perception of reality.
The problem is most acute when people are trying to summarise large areas of research using techniques like meta-analysis. In the hypothetical case of management practice x and species y from earlier, without including unpublished studies we could overestimate the average effects of management treatments. Although meta-analysis is great and I love what you can do with it, this is a fatal flaw.
The Centre for Evidence-Based Conservation (CEBC), who are the authority on systematic reviews and meta-analysis in ecology, recommend that researchers should hunt for non-published work to improve their analysis. While I agree that this is vastly preferable to including only studies from ISI journals, it still doesn’t solve the underlying problem. Contrary to the way scientists normally think we should actually be encouraging publication of negative results.
So how could we do this? A few journals which deal with applied subjects are already targeting the problem. The journal Restoration Ecology now has a section called “Set-backs and Surprises’’ which explicitly aims to publish negative results. As Richard Hobbs says in his editorial for the journal these results are just as important as hearing about projects which have worked. The website Conservation Evidence also aims to publish results, negative or positive, of conservation management in short, easy to understand articles. This should become more widespread outside of these areas. Synthesis of results is important for testing theory and the more information we have to test these theories the better.
Some people will undoubtedly read this and say “Hang on a minute! Surely positive results indicate good study design? We should only be considering the best research for testing theory or looking at the consequences of management.” Frankly these people can jump off a cliff. Yes, studies with near infinite sample sizes will find a difference between group A and group B. However, these differences will solely be a product of sample size. Ecological significance of effects is not the same as statistical significance. Yes, some studies with smaller sample sizes will have noisier results but we can account for this. The best means of testing a theory are by using as many different methodologies in as many different settings as possible. That is the true test of whether a theory fits. By excluding negative results we are, at best, slowing scientific progress. Given the pressures our natural world is facing, we do not have time for this.
Do you have any ideas how we could improve the biases in the ecological literature? How could we encourage publication of negative results, given they are generally perceived as less interesting?
Please feel free to leave any thoughts below.