Adventures in Signal Processing and Open Science

Tag: open peer review

It’s all about replication

ReScience logoA new journal appeared recently in the scientific publishing landscape: ReScienceannounced at the recent EuroSciPy 2015 conference. The journal has been founded by Nicolas Rougier and Konrad Hinsen. This journal is remarkable in several ways, so remarkable in fact that I could not resist accepting their offer to become associate editor for the journal.

So how does this journal stand out from the crowd? First of all it is about as open as it gets. The entire publishing process is completely transparent – from first submission through review to final publication. Second, the journal platform is based entirely on GitHub, the code repository home to a plethora of open source projects. This is part of what enables the journal to be so open about the entire publishing process. Third, the journal does not actually publish original research – there are plenty of those already. Instead, ReScience focuses entirely on replications of already published computational science.

As has been mentioned by numerous people before me, when dealing with papers based on computational science it is not really enough to review the paper in the classical sense to ensure that the results can be trusted (this not only a problem of computational science, but this is the particular focus of ReScience). Results need to be replicated to validate them and this is what ReScience addresses.

Many of us probably know it: we are working on a new paper of our own and we need to replicate the results of some previous paper that we wish to compare our results against. Except for that comparison, this is essentially lost work after you get your paper published. Others looking at the original paper whose results you replicated may not be aware that anyone replicated these results. Now you can publish the replication of these previous results as well and get credit for it. At the same time you benefit the authors of the original results that you have replicated by helping validate their research.

The process of submitting your work to ReScience is described on their website along with the review process and the roles of editors and reviewers. So if you have replicated someone else’s computational work, go ahead and publish it in ReScience. If it is in the signal processing area I will be happy to take your submission through the publishing process.

Open Access Journals: What’s Missing?

I just came across this blog post by Nick Brown: Open Access journals: what’s not to like? This, maybe… That post was also what inspired the title of my post. His post really got me into writing mode, mostly because I don’t quite agree with him. I left this as a comment on ihs blog, but I felt it was worth repeating here.

Read the rest of this entry »

Open review in the wild

Few journals and conferences so far seem to use open review. We mostly see open review practised as post-publication commenting on for example where it so far seems to be mainly about spotting errors in already published papers.

I would personally like to see more open review employed by journals and conferences in the publishing of scientific papers to increase transparency in the process.
Today I have found such an example thanks to Igor Carron’s post The papers for ICLR 2015 are now open for discussion! The machine learning conference International Conference on Learning Representations uses an open review model where reviews are published, anyone can comment on the papers, and anyone can ask to become a designated reviewer:

Even though independent sites exist for post-publication commenting and review, I think it is especially exciting to see it being actively encouraged and fully integrated into the paper submission and acceptance process by the conference organisers. In addition to providing transparency in the process, I hope it also stimulates more discussion when the it is actively encouraged as we see here.

How to attract reviewers for open / post-publication review?

In traditional journals with closed pre-publication peer review, reviewers are typically invited by the editor. Editors can for example draw on previous authors from the journal or (I guess) their professional network in general. With the recent appearance of several open peer review platforms, for example PubPeer, Publons etc. – more here, there will be a need to attract reviewers to such platforms. Sufficiently flawed papers seem to attract enough attention to trigger reviews, but it is my impression that papers that are generally OK do not get a lot of post-publication review. This is perhaps not such a big deal for papers that have been already been published in some journal with closed pre-publication review – the major downside is that the rest of us do not get to see the review comments. But, if you are going to base an entire journal on post-publication peer review you will want to ensure at least a few reviews of each published paper as a sort of stamp of approval.

The Winnower – the new journal based entirely on open post-publication peer review that I have previously written about here – is about to launch. First and foremost they will of course need to attract some papers to publish. Their in my opinion very fair (when you compare to other open access journals) price of $100 should help and the fact that they span a very broad range of scientific disciplines should also give them a lot of potential authors. They also want to attract reviewers to their papers. The platform is open to review by anyone, but in order to ensure a minimum number of reviews of each paper with at least some experts on the topic among them, this seems like a good idea. But how do you do this? As a new journal there are no previous authors to draw on. It can probably get difficult to get in touch with sufficiently many qualified reviewers across all of the journal’s disciplines and on top of that, reviewers may be more reluctant to accept since the reviews will be open with reviewers’ identities disclosed.

What can be done to attract sufficiently many reviewers? Should the journal gamble on being the cool new kid in class that everyone wants to be friends with or simply try to buy friends? I have been discussing this with their founder Josh Nicholson. One possibility is to pay reviewers a small amount for each review they complete. If we were talking one of the traditional publishers, whom I think in many cases are exploiting authors and reviewers shamelessly to stuff their own pockets, I think it would only be reasonable to actually start paying the reviewers. In the case of The Winnower, it may be different. The Winnower is a new journal trying to get authors and reviewers on board. With a very idealistic approach and pricing, I do not think people are likely to think that they are just trying to make money – being a “predatory publisher”. But on the other hand, paying reviewers might somehow make it look like they are trying to “buy friends”. With the journal’s profile, potential reviewers might mainly be ones that like to think of themselves as a bit idealistic and revolutionary too and that might just not go well with being paid for reviews? Josh told me an anecdote he had been told recently:

To illustrate this point, imagine walking down the street and an able bodied young man asks for your help loading a large box into a truck.  If he were to politely ask for help, most people would be highly likely to assist him in his request.  However, if he were to politely ask and also mention that for your time, he will pay you $0.25 most people will actually turn him down.  Despite being totally irrational, given that under the same circumstance they would do it without the promise of a quarter, the mention of money evokes the passerby to calculate what their time is worth to them.  To many, the $0.25 isn’t going to be worth the effort.  I think this may pose a similar issue.

Then again, paying reviewers could also send the message that they are taking their reviewers and the work they do very seriously? Ideally, I think it might work better with an incentive structure and “review of reviews” / scoring of reviews like the Stackexchange network for example. I am just afraid that something like that will take considerable “critical mass” to be effective. Another option Josh mentioned could be to let reviewers earn free publications with the journal by completing a number of reviews. This sounds better to me: you do not risk offending potential reviewers with a “price on their head”, but there is still something to gain for reviewers.

I think this is a very interesting question and probably one that a lot of people have much more qualified answers for than I. Let me know what you think?!

Forest Vista

seeking principles

Academic Karma

Re-engineering Peer Review

Pandelis Perakakis

experience... learn... grow


computing with space | open notebook


Peer-review is the gold standard of science. But an increasing number of retractions has made academics and journalists alike start questioning the peer-review process. This blog gets underneath the skin of peer-review and takes a look at the issues the process is facing today.

Short, Fat Matrices

a research blog by Dustin G. Mixon

Discover and manage research articles...

Science Publishing Laboratory

Experiments in scientific publishing

Open Access Button

Push Button. Get Research. Make Progress.

Le Petit Chercheur Illustré

Yet Another Signal Processing (and Applied Math) blog