Adventures in Signal Processing and Open Science

Peer Evaluation of Science

This is a proposal for a system for evaluation of the quality of scientific papers by open review of the papers through a platform inspired by StackExchange. I have reposted it here from The Self-Journal of Science where I hope my readers will go and comment on it: The proposal is also intended as a contribution to #peerrevwk15 on Twitter.

I have chosen to publish this proposal on SJS since this is a platform that comes quite close to what I envision in this proposal.


Researchers currently rely on traditional journals for publishing their research. Why is this? you might ask. Is it because it is particularly difficult to publish research results? Perhaps 300 years ago, but certainly not today where anyone can publish anything on the Internet with very little trouble. Why do we keep publishing with them, then? – they charge outrageous amounts for their services in the form of APCs from authors or subscriptions from readers or their libraries. One of the real reasons, I believe, is prestige.

The purpose of publishing your work in a journal is not really to get your work published and read, but it is to prove that your paper was good enough to be published in that particular journal. The more prestigious the journal, the better the paper, it seems. This roughly boils down to using the impact factor of the journal to evaluate the research of authors publishing in it (bad idea, see for example Wrong Number: A closer look at Impact Factors). It is often mentioned in online discussions how researchers are typically evaluated by hiring committees or grant reviewers based which journals they have published in. In Denmark (and Norway – possibly other countries?), universities are even getting funded based on which journals their researchers publish in.

I think the journal’s reputation (impact factor) is used in current practice because it is easy. It is a number that a grant reviewer or hiring committee member can easily look up and use to assess an author without having to read piles of their papers on which they might have to be experts. I support a much more qualitative approach based on the individual works of the individual researcher. So, to have any hope of replacing this practice, I think we need to offer a quantitative “short-cut” that can compete with the impact factor (and H-index etc.) that say little about the actual quality of the researcher’s works. Sadly, a quantitative metric is likely what hiring committees and grant reviewers are going to be looking at. Here I think a (quantitative) “score” or several such scores on different aspects of a paper accompanying the (qualitative) review can be used to provide such an evaluation metric. Here I am going to present some ideas of how such a metric can be calculated and also some potential pitfalls we need to discuss how to handle.

I believe that a system to quantify various aspects of a paper’s quality as part of an open review process could help us turn to a practice of judging papers and their authors by the merits of the individual paper instead of by the journal in which they are published. I also believe that this can be designed to incentivise participation in such a system.

Research and researchers should be evaluated directly by the quality of the research instead of indirectly through the reputation of the journals they publish in. My hope is to base this evaluation on open peer review, i.e. the review comments are open for anyone to read along with the published paper. Even when a publisher (in the many possible incarnations of that word) chooses to use pre-publication peer review, I think that should be made open in the sense that the review comments should be open for all to read after paper acceptance. And in any case, I think it should be supplemented by post-publication peer review (both open in the sense that they are open to read and also open for anyone to comment – although one might opt for a restriction of reviewers to any researcher who has published something themselves as for example Science Open uses).

What do I mean by using peer review to replace journal reputation as a method of evaluation? This is where I envision calculating a “quality” or “reputation” metric as part of the review process. This metric would be established through a quality “score” (could be multiple scores targeting different aspects of the paper) assigned by the reviewers/commenters, but endorsed (or not) by other reviewers through a two-layer scoring system inspired by the reputation metric from StackExchange. This would, in my opinion, comprise a metric that:

  1. specifically evaluates the individual paper (and possibly the individual researcher through a combined score of her/his papers),
  2. is more than a superficial number – the number only accompanies a qualitative (expert) review of the individual paper that others can read to help them assess the paper,
  3. is completely transparent – accompanying reviews/comments are open for all to read and the votes/scores and the algorithm calculating a paper’s metric is completely open.

I have mentioned that this system is inspired by StackExchange. Let me first briefly explain what StackExchange is and how their reputation metric works: StackExchange is a question & answer (Q&A) site where anyone can post questions in different categories and anyone can post answers to those questions. The whole system is governed by a reputation metric which seems to be the currency that makes this platform work impressively well. Each question and each answer on the platform can be voted up or down by other users. When a user gets one of his/her questions or answers voted up, the user’s reputation metric increases. The score resulting from the voting helps rank questions and answers so the best ones are seen at the top of the list.

The System

A somewhat similar system could be used to evaluate scientific papers on a platform designed for the purpose. As I mentioned, my proposal is inspired by StackExchange, but I propose a somewhat different mechanism as the one based on questions and answers on StackExchange does not exactly fit the purpose here. I propose the following two-layer system.

  • First layer: each paper can be reviewed openly by other users on the platform. When someone reviews a paper, along with submission of the review text, the reviewer is asked to score the paper on one or more aspects. This could be simply “quality”, whatever this means, or several aspects such as “clarity”, “novelty”, “correctness”. It is of course an important matter to determine these evaluation aspects and define what they should mean. This is however a different story and I focus on the metric system here.
  • Second layer: other users on the platform can of course read the paper as well as the reviews attached to it. These users can score the individual reviews. This means that some users, even if they do not have the time to write a detailed review themselves, can still evaluate the paper by expressing whether they agree or disagree with the existing reviews of the paper.
  • What values can a score take? We will get to that in a bit.

How are metrics calculated based on this two-layer system?

  • Each paper’s metric is calculated as a weighted average of the scores assigned by reviewers (first layer). The weights assigned to the individual reviews are calculated from the scores other users have assigned to the reviews (second layer). The weight could be calculated in different ways depending on which values scores can take. It could be an average of the votes. It could also be calculated as the sum of votes on each review, meaning that reviews with lots of votes would generally get higher weights than reviews with few votes.
  • Each author’s metric is calculated based on the scores of the author’s papers. This could be done in several ways: One is a simple average; this would not take into account the number of papers an author has published. Maybe it should, so the sum of scores of the author’s papers could be another option. Alternatively, it might also be argued that each paper’s score in the author’s metric should be weighted by the “significance” of the paper which could be based on the number of reviews and votes on these each paper has.
  • Each reviewer’s metric is calculated based on the scores of her/his reviews in a similar way to the calculation of authors’ metrics. This should incentivise reviewers to write good reviews. Most users on the proposed platform will act as both reviewers and authors and will therefore have both a reviewer and an author metric.

Which Values Can Votes Have?

I propose to make the scores of both papers (first layer) and individual reviews (second layer) a  ± 1 vote. One could argue that this is a very coarse-grained scale, but consider the option of for example a 10-level scale. This could cause problems of different users interpreting the scale differently. Some users might hardly ever use the maximum score while other users might give the maximum score to all papers that they merely find worthy of publication. By relying on a simple binary score instead, an average over a (hopefully) high number of reviews and review endorsements/disapprovals would be less sensitive to individual interpretations of the score value than many-level scores.


As mentioned, I hope the proposed model of evaluating scientific publications by accompanying qualitative reviews by a quantitative score would provide a useful metric that – although still quantitative – could prove a more accurate measure of quality of individual publications for those that need to rely on such a measure. This proposal should not be considered a scientific article itself, but I hope it can be a useful contribution to a debate on how to make peer review both more open and more broadly useful to readers and evaluators of scientific publications.

I have chosen to publish this proposal on SJS since this is a platform that comes quite close to what I envision in this proposal. I hope that readers will take the opportunity to comment on the proposal and help start a discussion about it.

It’s all about replication

ReScience logoA new journal appeared recently in the scientific publishing landscape: ReScienceannounced at the recent EuroSciPy 2015 conference. The journal has been founded by Nicolas Rougier and Konrad Hinsen. This journal is remarkable in several ways, so remarkable in fact that I could not resist accepting their offer to become associate editor for the journal.

So how does this journal stand out from the crowd? First of all it is about as open as it gets. The entire publishing process is completely transparent – from first submission through review to final publication. Second, the journal platform is based entirely on GitHub, the code repository home to a plethora of open source projects. This is part of what enables the journal to be so open about the entire publishing process. Third, the journal does not actually publish original research – there are plenty of those already. Instead, ReScience focuses entirely on replications of already published computational science.

As has been mentioned by numerous people before me, when dealing with papers based on computational science it is not really enough to review the paper in the classical sense to ensure that the results can be trusted (this not only a problem of computational science, but this is the particular focus of ReScience). Results need to be replicated to validate them and this is what ReScience addresses.

Many of us probably know it: we are working on a new paper of our own and we need to replicate the results of some previous paper that we wish to compare our results against. Except for that comparison, this is essentially lost work after you get your paper published. Others looking at the original paper whose results you replicated may not be aware that anyone replicated these results. Now you can publish the replication of these previous results as well and get credit for it. At the same time you benefit the authors of the original results that you have replicated by helping validate their research.

The process of submitting your work to ReScience is described on their website along with the review process and the roles of editors and reviewers. So if you have replicated someone else’s computational work, go ahead and publish it in ReScience. If it is in the signal processing area I will be happy to take your submission through the publishing process.

Open Access Journals: What’s Missing?

I just came across this blog post by Nick Brown: Open Access journals: what’s not to like? This, maybe… That post was also what inspired the title of my post. His post really got me into writing mode, mostly because I don’t quite agree with him. I left this as a comment on ihs blog, but I felt it was worth repeating here.

Read the rest of this entry »

Open review in the wild

Few journals and conferences so far seem to use open review. We mostly see open review practised as post-publication commenting on for example where it so far seems to be mainly about spotting errors in already published papers.

I would personally like to see more open review employed by journals and conferences in the publishing of scientific papers to increase transparency in the process.
Today I have found such an example thanks to Igor Carron’s post The papers for ICLR 2015 are now open for discussion! The machine learning conference International Conference on Learning Representations uses an open review model where reviews are published, anyone can comment on the papers, and anyone can ask to become a designated reviewer:

Even though independent sites exist for post-publication commenting and review, I think it is especially exciting to see it being actively encouraged and fully integrated into the paper submission and acceptance process by the conference organisers. In addition to providing transparency in the process, I hope it also stimulates more discussion when the it is actively encouraged as we see here.

Publishing mathematics in ebooks – part 1

This is the first part of what I hope will be a series of posts on my explorations of how to author maths-heavy writing in ebook format.

I have for quite some time now been annoyed with PDFs on mobile phones and tablets. Although there are some fine PDF viewers avaible, it usually still takes a lot of annoying scrolling to read a scientific paper on my phone or tablet. On the other hand, I have recently read a few novels as ebooks on my phone and my tablet and this has been an entirely different, enjoyable experience. The main difference is that the text in ebooks is re-flowable so as to make it easily adaptable to the screen size and preferred font size. This makes ebooks seem like a promising choice as an alternative to PDF for distributing scientific papers in more screen-friendly format. There is just one hurdle: mathematicsRead the rest of this entry »

Should we pay reviewers for their work?

I have previously discussed paying reviewers for their work. Although that was in the slightly different context of attracting reviewers for open post-publication peer review, a new open access journal is now introducing this idea in their workflow:

They do this by assigning reviewers and editors points for each paper they handle. A part of the APC of accepted papers goes into a pool and the accumulated points are then used as a basis of distribution to determine how large a bite of the cake each individual is payed. Editors and reviewers may then choose to keep the money, give the money back to the journal’s APC waiver pool, or donate it to their own university’s open access payments.

The journal has taken steps to ensure that this does not lead to inflation in the number of accepted papers just to earn points; editors and reviewers are assigned points for handling papers regardless of whether they are eventually accepted. Another IMO appealing feature of the journal is that reviews can be open if both authors and reviewers agree to this.

I am looking forward to seeing how this goes…

Workshop on Compressed Sensing in Wireless Communication

Qi Zhang, Jacek Pierzchlewski, and I (Thomas Arildsen) are organising a workshop on Compressed Sensing in Wireless Communication on May 22, 2015. The workshop is part of the conference European Wireless 2015 in Budapest, Hungary. Please see the workshop webpage for details on submission etc.

Teaching with the IPython Notebook

I have been teaching introductory Python for modelling and simulation and for scientific computing for a couple of years now. I am still somewhat new to Python myself, having “converted” from Matlab a couple of years ago. I find the open approach of using free and open source software instead of expensive proprietary software very motivating and I was easily talked into using it by my colleagues and quickly decided to base my teaching on it as well.
Read the rest of this entry »

Magni: A Python Package for Compressive Sampling and Reconstruction of Atomic Force Microscopy Images

Our new software metapaper Magni: A Python Package for Compressive Sampling and Reconstruction of Atomic Force Microscopy Images has just been published in Journal of Open Research Software. The paper describes our new software package Magni:

Magni is an open source Python package that embraces compressed sensing and Atomic Force Microscopy (AFM) imaging techniques. It provides AFM-specific functionality for undersampling and reconstructing images from AFM equipment and thereby accelerating the acquisition of AFM images. Magni also provides researchers in compressed sensing with a selection of algorithms for reconstructing undersampled general images, and offers a consistent and rigorous way to efficiently evaluate the researchers own developed reconstruction algorithms in terms of phase transitions. The package also serves as a convenient platform for researchers in compressed sensing aiming at obtaining a high degree of reproducibility of their research.

The software itself is on GitHub as well as on Aalborg University’s repository: DOI 10.5278/VBN/MISC/Magni

Go ahead and check it out if you are into compressed sensing or atomic force microscopy. Pull requests welcome if you have ideas.

Live-tweeting iTWIST 2014 workshop

As an experiment I am live-tweeting the workshop iTWIST in Namur, Belgium. Look for the tag #itwist14.
See also for inspiration (by @collabchem and @eperlste)

Academic Karma

Re-engineering Peer Review

Pandelis Perakakis, PhD

Academic Website


computing with space | open notebook


Peer-review is the gold standard of science. But an increasing number of retractions has made academics and journalists alike start questioning the peer-review process. This blog gets underneath the skin of peer-review and takes a look at the issues the process is facing today.

Short, Fat Matrices

a research blog by Dustin G. Mixon

Discover and manage research articles...

Science Publishing Laboratory

Experiments in scientific publishing

Open Access Button

Push Button. Get Research. Make Progress.

Le Petit Chercheur Illustré

Yet Another Signal Processing (and Applied Math) blog


Get every new post delivered to your Inbox.

Join 292 other followers