Can bad reviews be useful?

Everyone who publishes has had bad experiences with peer review. Reviewers that miss the point of what you are trying to say, or just hate your paper.

Case in point is some work on tropical logging we just got published in Forest Ecology and Management (see my blog post on it here and the paper here). I don’t have loads of experience with peer review, but now have 3 papers under my belt, one currently in review and I have reviewed about 10 papers in total. One of the reviews I got for this paper was the worst I have ever been given. I’m not going to go into detail, but here are some choice quotes:

Unfortunately, This analysis does not bring any new results in comparison with others recently published synthesis…

Finally the assess of the impact of logging on tree species richness presented in this study is meaningless…

The straight forward conclusion of the authors does not bring much to the debate already closed….

Now. The the first two comments may or may not be true. The thing that annoyed me more than anything was the last comment. Debate in science should never be closed. Scientists should not provide a united front when there is contradictory evidence. If you disagree with a study either (a) re-analyse the data of the paper you don’t like and write a letter to the editor, (b) produce another piece of work testing the same hypotheses or (c) be really radical and offer to write a joint paper with the authors of the work you disagree with. Blocking the paper from publication should not be an option.

The review I got is nothing compared to what some others have had to deal with, but it was annoying to have my path blocked by a reviewer who didn’t want my work published simply because my results didn’t fit their world view.

However, I have taken some positives lessons from this. Firstly, try to be aware of your own biases when reviewing someone else’s work. Secondly, be fair and be careful with what you say. If you don’t like a paper don’t go on and on about it, remember that someone spent months of their life on that work. We constructive and concise. Thirdly, when you are writing a paper don’t go out of your way to be controversial. I think some of our drafts of the paper came off as a bit combative and thus produced this reaction. Getting this reaction from a reviewer probably means that some readers will have similar reactions. However, don’t shy away from controversy either. If your results support a controversial hypothesis don’t let people who disagree with your view of things block you from publication.

Just what is resilience, anyway?

Last week I organised a workshop bringing together researchers interested in resilience from across the Biodiversity and Ecosystem Service Sustainability (BESS) programme run by NERC. I’ll write about things that came out of it over the next few weeks. Here is my first missive

“Is it real? Is it an obscure object of desire?” my boss Adrian asked during our workshop . Given that  nearly 20 years ago there were 163 different definitions of ecosystem resilience, it is perhaps no wonder that we we having a few problems. Part of this problem is that resilience is a boundary object – a term that is interpreted differently by different communities. During our meeting it became clear, for example, that ecological researchers and policy-makers did not necessarily mean the same thing when they were talking about resilience.

Generally, researchers see resilience as a property of a system. Being researchers, we want to quantify this resilience. However, it turns out that resilience can’t really be viewed as a single thing, since it is made up of a number of different qualities. Along with the expert guidance of Volker Grimm our workshop came up with 3 different elements that are important when assessing resilience for research:

  1. Recovery – The return of a variable to the reference state after a disturbance.
  2. Resistance – A variable staying essentially unchanged despite disturbances.
  3. Persistence – Persistence of the system over time.

Using these three different properties allows researchers to look at different aspects of resilience and compare across systems. Making such comparisons is actually very difficult due to constrains on time and funding, as well as logistical problems. For example, to compare resistance of different communities you would ideally apply different intensities of disturbance in different locations. This may be possible in some relatively ‘fast’ systems such as grasslands but it is unlikely that you would get permission to do this to a woodland where you might have to cut trees down. In order to resolve this, mechanistic models can be exceptionally useful for investigating different scenarios of change. Combining this with empirical data collection in the same system can help us gain a more detailed understanding of resilience. This is something we are aiming to do in our current project as part of my post-doc work.

Policy makers on the other hand generally view resilience as a goal. Recently policy documents have begun to mention the importance of resilience. For example, one of the Convention on Biological Diversity’s 2020 aims is to:

By 2020, ecosystem resilience and the contribution of biodiversity to carbon stocks has been enhanced, through conservation and restoration, including restoration of at least 15 per cent of degraded ecosystems, thereby contributing to climate change mitigation and adaptation and to combating desertification.

Similarly the environment white paper in published by the UK government in 2010 mentions resilience 36 times and the Welsh government is aiming to create:

A biodiverse natural environment with healthy functioning ecosystems that support social, economic and ecological resilience and the capacity to adapt to change.

It is also included in US and Australian policy. So in the case of policy-makers it becomes clear that resilience is seen as a target. While for researchers resilience can mean something very specific policy-makers probably consider it to be closest to the previous definition of persistence.  At our workshop there were plenty of anecdotes about policy-makers saying things like “resilience is the new sustainability” and telling civil servants to “stick some resilience in your report, it’s the new thing.” There were also reports that some policy-makers wanted the production of maps of resilience. I think this is potentially dangerous. Given that the ratio of empirical work to conceptual stuff/reviews and perspectives pieces is about 1:1000 we simply don’t have enough evidence to produce these maps at the moment. If push came to shove then we could probably come up with a best guess based on ecological theory, but even then there would be all sorts of caveats.

I think it’s clear we will never reach a point where there is one definition of resilience that fits everyone’s need. However, when we talk about resilience we need to be clearer about what we mean by it. So next time you use it in a paper, for the love of god, define it.

*Edit #1 – I just came across this nice post by Jeremy Fox on defining stability concepts in ecology, which, if might be a useful companion piece to what I said here.

*Edit #2 – Ambroise Baker who helped organise the workshop with me has a short summary of the meeting over on the Lake BESS blog, you can see that here.

Fear of commitment

This picture has nothings to do with the post – apart from the title.

Last week I went to a fantastic workshop run by the British Ecological Society. This blog has almost nothing to do with that but concerns a conversation I had in the pub afterwards. I will not name any of the people involved since it doesn’t seem fair.

After the workshop we went to an agreeable pub that only slightly resembled those from the film “The end of the world.” On the plus side it had a reasonable selection of drinks, though you did have to re-mortgage your house to buy one.

While there I was discussing post-PhD career options with people. As far as I remember the conversation went a bit like this:

Me: “The lack of job security in academia scares me a bit.”

Person A : “Yeah it’s tough, I worked abroad for a post-doc and my wife and I found it really hard to live in a place I didn’t like to further my career”

Me: “For me it’s more the knowledge that most PhD students will never get a permanent academic position and that we might have to work 60 or 70 hour weeks for the best part of 10 years to get close. Personally that’s why I think it’s always worth having a ‘Plan B.’”

Person B: “But if you have a ‘Plan B’ you might not fully commit to the job. I mean look at [name redacted] it’s inspiring, she lives, eats and breathes science. That’s her life.”

Now, this last bit stuck with me.

Is that what people think?

That to be a committed scientist means you neglect all other things in your life? That you shouldn’t have children or even hobbies because they get in the way? That without that you shouldn’t be doing the job?

There was a lot of talk at the workshop about the fact that having a career break to have a family would not be seen by funding agencies as a bad thing. That’s great in practice, but if other people hold the same attitude as ‘Person B’ what’s the point? Having a child is surely more of a distraction than me thinking that I might want to switch careers at some point in the future? If these people sit on funding boards we should just give up talking about diversity in academia because it’s never going to happen.

My lack of confidence that I will get a permanent job in academia is not unfounded, it comes from personal experience and more importantly given that I am allegedly a scientist, from real statistical evidence.

I have a few friends that are now well into their scientific careers and have about 10 years post-doc experience. That puts them in their late 30’s. These friends have young families and have moved countries a few times now. They have all published well and have a Nature papers between them. Yet they are facing the prospect of being ‘retired’ by science because they can’t get a permanent position and they are too expensive to hire as post-docs because of the amount of experience they have.

Getting out of science when you’ve reached that point is much more traumatic than doing it early on. Maybe a ‘Plan B’ would have been a good idea for these friends. What they’re going through is really fucking hard.

Reading other blogs has made me acutely aware that this is not a problem unique to my friends. The proportion of PhD students that get a permanent position has been decreasing as the number of PhDs rises. Because most of the people who run things – invariably white, middle-aged men – got their jobs as a time when it was easier to get a permanent position they don’t recognise this problem. Better people than me have said previously that PhD training should acknowledge this and emphasise the ways in which our skills can be used outside of traditional academic roles – that is after all where most of us will end up, whether we like it or not.


Sorry, if that was a bit of a rambly blog post this week but I needed to get this off my chest. Please respond with your thoughts. Do you have experience of switching to a non-research job after a PhD? Or have you found it easy to hold down a job in research whilst having a satisfying life outside of work? Either way let me know.