Adventures in Signal Processing and Open Science

In traditional journals with closed pre-publication peer review, reviewers are typically invited by the editor. Editors can for example draw on previous authors from the journal or (I guess) their professional network in general. With the recent appearance of several open peer review platforms, for example PubPeer, Publons etc. – more here, there will be a need to attract reviewers to such platforms. Sufficiently flawed papers seem to attract enough attention to trigger reviews, but it is my impression that papers that are generally OK do not get a lot of post-publication review. This is perhaps not such a big deal for papers that have been already been published in some journal with closed pre-publication review – the major downside is that the rest of us do not get to see the review comments. But, if you are going to base an entire journal on post-publication peer review you will want to ensure at least a few reviews of each published paper as a sort of stamp of approval.

The Winnower – the new journal based entirely on open post-publication peer review that I have previously written about here – is about to launch. First and foremost they will of course need to attract some papers to publish. Their in my opinion very fair (when you compare to other open access journals) price of $100 should help and the fact that they span a very broad range of scientific disciplines should also give them a lot of potential authors. They also want to attract reviewers to their papers. The platform is open to review by anyone, but in order to ensure a minimum number of reviews of each paper with at least some experts on the topic among them, this seems like a good idea. But how do you do this? As a new journal there are no previous authors to draw on. It can probably get difficult to get in touch with sufficiently many qualified reviewers across all of the journal’s disciplines and on top of that, reviewers may be more reluctant to accept since the reviews will be open with reviewers’ identities disclosed.

What can be done to attract sufficiently many reviewers? Should the journal gamble on being the cool new kid in class that everyone wants to be friends with or simply try to buy friends? I have been discussing this with their founder Josh Nicholson. One possibility is to pay reviewers a small amount for each review they complete. If we were talking one of the traditional publishers, whom I think in many cases are exploiting authors and reviewers shamelessly to stuff their own pockets, I think it would only be reasonable to actually start paying the reviewers. In the case of The Winnower, it may be different. The Winnower is a new journal trying to get authors and reviewers on board. With a very idealistic approach and pricing, I do not think people are likely to think that they are just trying to make money – being a “predatory publisher”. But on the other hand, paying reviewers might somehow make it look like they are trying to “buy friends”. With the journal’s profile, potential reviewers might mainly be ones that like to think of themselves as a bit idealistic and revolutionary too and that might just not go well with being paid for reviews? Josh told me an anecdote he had been told recently:

To illustrate this point, imagine walking down the street and an able bodied young man asks for your help loading a large box into a truck.  If he were to politely ask for help, most people would be highly likely to assist him in his request.  However, if he were to politely ask and also mention that for your time, he will pay you $0.25 most people will actually turn him down.  Despite being totally irrational, given that under the same circumstance they would do it without the promise of a quarter, the mention of money evokes the passerby to calculate what their time is worth to them.  To many, the $0.25 isn’t going to be worth the effort.  I think this may pose a similar issue.

Then again, paying reviewers could also send the message that they are taking their reviewers and the work they do very seriously? Ideally, I think it might work better with an incentive structure and “review of reviews” / scoring of reviews like the Stackexchange network for example. I am just afraid that something like that will take considerable “critical mass” to be effective. Another option Josh mentioned could be to let reviewers earn free publications with the journal by completing a number of reviews. This sounds better to me: you do not risk offending potential reviewers with a “price on their head”, but there is still something to gain for reviewers.

I think this is a very interesting question and probably one that a lot of people have much more qualified answers for than I. Let me know what you think?!