Openness And Anonymity in Peer Review

by ThomasA

About a month ago, I attempted to give an overview of a few open review platforms. In relation to that, the question of anonymity of the reviewers came up. I have given it some more thought and would like to discuss some of these thoughts.

First of all, when I talk about open/closed and anonymous/identified review, I would like to point out that I consider these two independent “dimensions” of the nature of scientific peer review:

Anonymous Identified
Closed 1 2
Open 3 4

So, reviews can either be closed or open and at the same time, anonymous or identified.

Traditional journals have, to the best of my knowledge, more or less all been in the first “quadrant” above, i.e. both closed in the sense that they are not available to the readers of the journal after publication; and anonymous in the sense that the authors cannot see who the reviewers are. I do not know any examples of publishing in the second quadrant. UPDATE: Matt Hodgkinson and Magnus Rattray were kind enough to give me two examples of review in the second quadrant as well. Matt Hodgkinson:

…reviewers at the BMJ have to sign their comments http://www.bmj.com/about-bmj/resources-authors/peer-review-process and this is optional at PLOS ONE http://www.plosone.org/static/reviewerGuidelines#anonymity. Neither publishes the reviews.

Magnus Rattray:

In the NIPS conference reviewers and programme chairs are identified to each other but anonymous to the authors. This introduces some advantages of openness (improved review quality, more chance to identify conflicts) while maintaining some advantages of anonymity (fear of reprisals or ruined friendships).

Publishing models and experiments are starting to pop up in the third and fourth quadrants – open review, where the reviewers are anonymous in some cases and in other cases identified.

Let’s take a look at the anonymity issue. I am not sure there is a clear answer to what is best, but there are clearly both advantages and drawbacks of anonymous resp. identified reviews that I think we need to discuss. Below I list a few advantages and drawbacks of anonymity where I assume that a drawback of anonymous review is an advantage of identified review and vice versa.
Drawbacks

  • Reviewers do not get credit for their work. They cannot, for example, reference particular reviews in their CVs as they can with publications.
  • It is relatively “easy” for a reviewer to provide unnecessarily blunt or harsh critique.
  • It is difficult to guess if the reviewer has any conflict of interest with the authors by being, for example, a competing researcher interested in stalling the paper’s publication.

Advantages

  • Reviewers do not have to fear “payback” for an unfavourable review that is perceived as unfair by the authors of the work.
  • Some (perhaps especially “high-profile” senior faculty members) reviewers might find it difficult to find the time to provide as thorough a review as they would ideally like to, yet would still like to contribute and can perhaps provide valuable experienced insight. They can do so without putting their reputation on the line.

What else am I missing?

I have put a publicly editable version of this list on Google Drive – feel free to add points if you like, or comment below: https://docs.google.com/document/d/1RMEHerNSpnR9yg98qfHv1hlKjvWznVGpDUPUGC_cFkg/edit?usp=sharing