For anyone who has undergone the process of having an article face peer-review – at a conference or at a journal – it should come to no surprise to them that there is some randomness in the process. We’ve all heard the stories: a paper that is rejected from one conference wins a top paper award at the next. A new study tests the influence that referee decisions make on the quality of accepted papers. Using a statistical-imputation method, they vary the number of “rational” referees at a journal – in other words, self-interested referees who refuse good work that would compete with their own – to determine the impact on quality. And as we might expect, the outcome is dramatic:

“Our message is clear: if it can not be guar- anteed that the fraction of ’rational’ and ’random’ refer- ees is confined to a very small number, the peer review system will not perform much better than by accepting papers by throwing (an unbiased!) coin.”

Hardly surprising, right? One critique offered to this study is that unbiased editors are meant to correct for some of these errors. But in some ways, editors are the ones who have the most opportunity to be biased. They can accept a paper, even without much reviewer support, or can reject a paper that reviewers approve. And while the blinding process is never perfect – the existence of conferences, in-text references and cues, and even the style of the writer can cue “blinded” reviewers into whose paper they are examining – the editors are the only people who actually know the identity of the researchers, giving them more information to make a “rational” decision.

While I don’t think – and certainly hope – that most reviewers do not act in their own self-interest in reviewing papers, it is telling that this is the behavior considered “rational.” So what can be done? The author of this study offers one suggestion: create a marketplace of papers, where journals can compete for interesting papers while the less sought-after papers have fewer options. This is an intriguing idea, but it does get rid of one of the benefits of peer review: the changes recommended by one’s peers often make a paper substantially better. That said – and in line with this study – these reviewers sometimes make critical errors and reject papers on the basis of a false reading or misunderstanding of the study – and it remains unfair that the paper authors have no recourse against such misunderstanding.

It is also interesting that this study only varies the number of “rational” reviewers, without closely examining also the number of “random” reviewers – the reviewers who cannot judge the paper’s quality sufficiently. Reviewing is hard work and reviewers can often end up passing judgment on a piece outside their area of expertise – which again inserts randomness in the process. It would be fascinating to have some study attempt to survey researchers to see how many feel like they’ve had to review a paper outside their expertise (random) or have let their personal goals influence a review (rational) – of course, only a truly anonymous study could produce even somewhat-reliable results (who wants to admit either to a lack of expertise or selfishness?).

Although we probably all agree that there are some changes required to the process of peer-review, let’s make sure these changes benefit the scholarly community. Getting rid of the revision process would be a mistake, but more accountability for reviewers certainly seems necessary.