Adventures in Signal Processing and Open Science

Openness And Anonymity in Peer Review

About a month ago, I attempted to give an overview of a few open review platforms. In relation to that, the question of anonymity of the reviewers came up. I have given it some more thought and would like to discuss some of these thoughts.

First of all, when I talk about open/closed and anonymous/identified review, I would like to point out that I consider these two independent “dimensions” of the nature of scientific peer review:

Anonymous Identified
Closed 1 2
Open 3 4

So, reviews can either be closed or open and at the same time, anonymous or identified.

Traditional journals have, to the best of my knowledge, more or less all been in the first “quadrant” above, i.e. both closed in the sense that they are not available to the readers of the journal after publication; and anonymous in the sense that the authors cannot see who the reviewers are. I do not know any examples of publishing in the second quadrant. UPDATE: Matt Hodgkinson and Magnus Rattray were kind enough to give me two examples of review in the second quadrant as well. Matt Hodgkinson:

…reviewers at the BMJ have to sign their comments http://www.bmj.com/about-bmj/resources-authors/peer-review-process and this is optional at PLOS ONE http://www.plosone.org/static/reviewerGuidelines#anonymity. Neither publishes the reviews.

Magnus Rattray:

In the NIPS conference reviewers and programme chairs are identified to each other but anonymous to the authors. This introduces some advantages of openness (improved review quality, more chance to identify conflicts) while maintaining some advantages of anonymity (fear of reprisals or ruined friendships).

Publishing models and experiments are starting to pop up in the third and fourth quadrants – open review, where the reviewers are anonymous in some cases and in other cases identified.

Let’s take a look at the anonymity issue. I am not sure there is a clear answer to what is best, but there are clearly both advantages and drawbacks of anonymous resp. identified reviews that I think we need to discuss. Below I list a few advantages and drawbacks of anonymity where I assume that a drawback of anonymous review is an advantage of identified review and vice versa.
Drawbacks

  • Reviewers do not get credit for their work. They cannot, for example, reference particular reviews in their CVs as they can with publications.
  • It is relatively “easy” for a reviewer to provide unnecessarily blunt or harsh critique.
  • It is difficult to guess if the reviewer has any conflict of interest with the authors by being, for example, a competing researcher interested in stalling the paper’s publication.

Advantages

  • Reviewers do not have to fear “payback” for an unfavourable review that is perceived as unfair by the authors of the work.
  • Some (perhaps especially “high-profile” senior faculty members) reviewers might find it difficult to find the time to provide as thorough a review as they would ideally like to, yet would still like to contribute and can perhaps provide valuable experienced insight. They can do so without putting their reputation on the line.

What else am I missing?

I have put a publicly editable version of this list on Google Drive – feel free to add points if you like, or comment below: https://docs.google.com/document/d/1RMEHerNSpnR9yg98qfHv1hlKjvWznVGpDUPUGC_cFkg/edit?usp=sharing

The Open Access Button

Recently, David Carroll and Joseph McArthur, medical resp. pharmacology students from London, came up with a great little idea: why not develop an easy way for people looking to read scientific papers online to report when they encounter a paper they cannot access, because they have to pay for it? That is, when they “hit a paywall”. David and Joseph set out to realise this idea by developing a browser button that users can click when that happens. The idea is to record reported incidents in a database to calculate statistics of how large this problem is. The idea has been well received and lots of people seem to have joined the effort to help develop it. You can read more about it here:

http://blogs.plos.org/thestudentblog/2013/08/20/if-someone-hits-a-paywall-in-the-forest-does-it-make-a-sound-the-open-access-button/

and follow their progress here:

http://oabutton.wordpress.com/.

Forest Vista

seeking principles

Academic Karma

Re-engineering Peer Review

Pandelis Perakakis

experience... learn... grow

chorasimilarity

computing with space | open notebook

PEER REVIEW WATCH

Peer-review is the gold standard of science. But an increasing number of retractions has made academics and journalists alike start questioning the peer-review process. This blog gets underneath the skin of peer-review and takes a look at the issues the process is facing today.

Short, Fat Matrices

a research blog by Dustin G. Mixon

www.rockyourpaper.org

Discover and manage research articles...

Science Publishing Laboratory

Experiments in scientific publishing

Open Access Button

Push Button. Get Research. Make Progress.

Le Petit Chercheur Illustré

Yet Another Signal Processing (and Applied Math) blog