Date: 17.02.2021
Has the classic peer review process had its day?
Peer reviews are most commonly used to determine an academic paper's suitability for publication. A peer review is essential for the assessment of the propriety of a scientific article prior to its publication and for guaranteeing its quality. The peer review procedure thus serves to ensure that scientific articles are subjected to critical review by experts before publication. The authors of the peer-reviewed work must take any criticism seriously and correct any errors discovered, or explain why the comments of the reviewers are inaccurate before the study can be published. Many scientific journals use peer review. Similarly, the quality of applications for funding of research projects is usually assessed by peer review as well. As far as the reputation of an article is concerned, an academic publisher or a scientific journal with peer-review procedures should always be preferred. However, the classic peer review process has its downsides. As much as it is popular as a means of establishing academic integrity, it is also flawed. New alternatives to the classic procedure are emerging as an opposition. In this article the peer review process will be explained in depth. Its advantages and disadvantages will be laid out. Furthermore, the criticism towards the traditional peer review process will be discussed. An alternative to the classic approach that is gaining popularity will be presented.
The classic peer review process: a dive into the procedure
The peer review is a useful tool in assuring that an article is appropriate for publication. Peer review involves subjecting the author's scholarly work and research to the scrutiny of other experts in the same field to check its validity and evaluate its suitability for publication. A peer review helps the publisher decide whether a work should be accepted. There are standard steps that are usually taken when the process of peer review comes about.
From submission to publication of papers
When a scholarly work is submitted to a scientific journal, it first undergoes a preliminary check known as a desk review. The editor decides if the manuscript should be sent for peer review or be immediately rejected. The next step is to select experts from the same field who are qualified and able to review the work impartially. Ideally the work is evaluated by multiple experts. However, the process usually goes as following:
- Submission of paper
The corresponding author submits their paper to the journal they have selected. This is usually done via an online system. Occasionally, journals may accept submissions by email.
- Editorial office assessment
The journal checks the paper’s composition and arrangement against their guidelines to make sure it includes the required sections and stylizations. The quality of the paper is not yet assessed at this point.
- Appraisal by the editor
Usually, the editor-in-chief at the journal checks whether the submitted article is suitable and captivating enough. If not, it may be rejected without being reviewed further.
- Invitation to reviewers
If the submitted article has been deemed appropriate by the editor at the academic journal, invitations are sent out to reviewers who are believed to be suitable. As responses are received, further invitations are issued, if necessary, until the required number of acceptances is obtained – commonly this is two, but there is some variation between journals.
- Response to invitations
Potential reviewers consider the invitation against their own expertise, conflicts of interest and availability. They then accept or decline. If possible, when declining, they might also suggest alternative reviewers.
- Reviews are conducted
The reviewer sets time aside to read the paper several times. The first read is used to form an initial impression of the work. If major problems are found at this stage, the reviewer may feel comfortable rejecting the paper without further work. Otherwise they will read the paper several more times taking notes so as to build a detailed point-by-point review. The review is then submitted to the journal with a recommendation to accept or reject it – or else with a request for revision (usually flagged as either major or minor) before it is reconsidered.
- The reviews are evaluated by the journal
The editor considers all the returned reviews before making an overall decision. If the reviews differ widely, the editor may invite an additional reviewer so as to get an extra opinion before making a decision.
- The decision is made
The editor sends a decision email to the author including any relevant reviewer comments. Whether the comments are anonymous or not will depend on the type of peer review that the journal operates. Consequently, the paper is sent to production and the reviewers are informed about the outcome of their review.
The process from submission of an article for peer review to its publication may vary depending on the journal, but in most cases these steps are followed. The peer review process is considered to be the most valid form of research evaluation and there are many reasons why it is chosen by scientists.
High-quality evaluation and academic integrity at the basis of the peer review
Peer review is a cornerstone of the process of bringing academic research to publication. Scientific findings and discoveries can have far-reaching implications for individuals and society. That is why a peer review is helpful in determining the quality of an academic article. The process serves as quality control prior to publication. What is more, peer reviews prove beneficial for authors, too. The filtering process and revision advice improve the quality of the final research article as well as offer the author new insights into their research methods and the results that they have compiled. Peer review gives authors access to the opinions of experts in the field who can provide support and insight. It ensures the integrity of the publishing process and the scholarly record. Reviewers are independent of journal publications and the research being conducted. Peer review is an effective form of research evaluation to help select the highest quality articles for publication.
There are several types of peer review. Different journals use different methods.
Types of peer review
The most common form of peer review in the field of science is the single blind. In this type of peer review the author does not know who the reviewers are, but the reviewers know who the author is. It is preferred because the anonymity allows the reviewer to be honest without fear of criticism from an author. Knowing who the author is (and their affiliation) allows the reviewer to use their knowledge of the author's previous research. In a single blind peer review, authors cannot contact the reviewer since they do not know who they are. This takes a considerable amount of pressure off reviewers and allows them to judge research more objectively. The best peer review is honest and unflinching. And sometimes, this means shining a bright light on a work’s limitations so everyone can see them. However, the single blind has some disadvantages. Knowledge of the author may overshadow the quality of the work – potentially leading to a lack of scrutiny, especially if it is the work of an author with a dazzling track record. A single-blind peer review does not protect the authors against gender, racial or geographic bias. There is the potential for discrimination based on gender or nationality. In the case of perceived discrimination on the basis of nationality it is often conflated with discrimination on the basis of bad language skills.
Another popular form of peer review is the double-blind peer review. In this type of peer review the reviewers do not know the identity of authors, and vice versa. Generally, if the author’s identity is unknown to the reviewer, it will prevent the reviewer from forming any bias. Since the reviewer does not know who wrote the paper, they cannot be influenced by their standing within a research community or even gender. Sometimes it can be difficult to write a bad review (even if justified) on someone who is well-known within a particular sector. By having a double-blind review, this is eliminated from happening. However, there are some arguments against the double-blind form of peer review. Many state that it is not really blind after all. With some research sectors being so small, it is very easy to have an educated guess of who the author is. When an author is trying to make a point in their research, they may self-reference from previous work they have written, and in some cases it becomes obvious who the author is. To ensure it is totally blind, an author would have to remove all references to themselves and their work in the paper, which could ruin the research.
In some cases, there could also be a triple-blind form of peer review where not only the identities of the author and reviewers are hidden from each other, but also from the editor. Currently, a few journals use this system, but it remains fairly uncommon in designing review processes.
The single-blind and double-blind forms of peer review are considered traditional approaches. In recent years, however, they have attracted their fair share of criticism.
The death of the classic peer review process? Criticism and scrutiny
There is little evidence on the effectiveness of peer review, but considerable evidence on its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud, the traditional peer review process is judged as slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused.
Too slow of a process
Many journals, even in the age of the internet, take more than a year to review and publish a paper. It is hard to get good data on the cost of peer review, particularly because reviewers are often not paid. Usually for the status of the process to be changed from “reviewers invited” to “under review”, it takes an average of two weeks to a month. Depending on the specific subject area of your study and the availability of the reviewers, this process may take different amounts of time to begin. For many academics, the time spent reviewing could be spent doing something more productive – like original research. It is known for a fact that peer review takes time – at least seven to eight hours per paper done properly, often with no remuneration or recognition for the reviewer and hence rarely regarded as a priority in a busy academic schedule. That takes away some of the motivation for the reviewer, which could result in a review that is not thought out and precise. And because peer reviews take time both for publishers and reviewers, there could potentially be delays in the author’s dissertation and research.
Neutrality is doubtful
One pretty significant problem with peer review is that it may be prone to bias from the reviewers. Not only are women greatly underrepresented in the peer review process, but reviewers are much more likely to have a preference to work done by those that are the same gender as themselves. This is obviously a serious issue, especially when science already has a problem with sexism. What is more, even in the 21st century there might be bias and prejudice based on race, ethnicity, nationality, culture and political views. The single-blind form of peer review certainly does not do much to protect the author against bias. The editorial peer review process has also been strongly biased in the past against “negative studies”, i.e. studies that find an intervention does not work.
Even if there is no discrimination against an author, neutrality is still an important factor that is often overlooked. In the single blind form of peer review, reviewers may often make their judgment taking into account the author’s previous body of work. Peer review bias is difficult to detect and quantify. Although the double-blind format of review now presents a way of combating referees’ bias, in the digital age, critics have repeatedly pointed to the ineffectiveness of blinding author identities as a simple Google-search commonly enables identifying the authors of a ‘blinded’ manuscript.
Who calls the shots
Much of the decision-making power rests in the hands of the editors, who are the link between author and reviewer. In most cases, this is fine, and helps the process to proceed smoothly, but it can lead to misuse of this authority. Editors often stand accused of arbitrarily rejecting manuscripts before they reach the peers. There is also evidence that decisions are often judged by country: for example, a US-based journal is much more likely to reject non-US papers, whatever the quality.
Falsifying reviews
One of the most common issues that affects reviewing negatively in the modern age stems from the practice of soliciting author-suggested reviewers. Although the intention may be good: identify a wider network of reviewers, especially for more specialist areas, by asking an obvious expert – the author, who should be ideally placed to suggest suitable authorities and perhaps save the editor a good deal of leg work –, the result is often not. The suggested reviewers often turn out to be fake, or if they are indeed real and appropriate authorities for the topic area, their email addresses end up leading back to the author. There is even evidence of agencies offering fake peer reviews to authors for a guaranteed positive outcome.
Manipulating citations and plagiarism
Citation manipulation is another malfeasance that causes havoc during the peer review process. Editors sometimes ask authors to cite more papers from their journal in an attempt to increase the overall citations in the journal. Another variant is when reviewers require authors to mass-cite their papers regardless of their relevance to the manuscript at hand.
A scandalous instance is when a reviewer seemingly rejects an article, but later submits their own paper based on a similar topic. This is obviously not an easy situation to spot (unless said reviewer was foolish enough to submit the second article to the same journal) but it could raise an eyebrow if a reviewer rejects an obviously decent paper.
It is obvious that the single-blind and double-blind forms of peer review both have their fair share of downsides. With an abundance of factors threatening the integrity of the classic peer review, it comes as no surprise that alternative options are gaining momentum in the academic world.
Saving grace of the peer review: the open peer review process
With the advancement of information and communication technologies, science is entering a new era. The abundance of online, open access data, software and article repositories now enables scholars to use and share information more efficiently than ever before. This has resulted in the transformation of traditional research, conducted by localized groups depending on their own resources and merits, to a more dynamic and globally interconnected effort, where ideas, data and scientific software are instantly accessible to the entire academic network. This transformation bears significantly positive consequences both for science and society. Free online access to academic material means that scientific research can now be built on firmer ground, be more visible, and have a greater impact. Open access to scientific knowledge also benefits society by allowing policy-makers, institutions, grant-awarding bodies and the general public to make better informed decisions.
This poses the question of where the peer review process stands. Traditional journal peer review has been repeatedly criticized as slow, unreliable and susceptible to bias or even fraud. There is a growing conviction among scholars that science and society would profit from the scrutiny of original scientific ideas and results by the entire academic community whose collective knowledge is likely to result in more accurate and objective evaluations. That is where the dynamic open peer review process emerges.
Change where change is needed: modifications of the open peer review
Open peer review process is the opposition to the classic single- and double-blind peer review. Open peer review is a term that describes various possible modifications of the traditional scholarly peer review process. There are three most common modifications that distinguish the open peer review from the traditional form.
- Open identities: both the author and reviewer are aware of each other’s identities, unlike in the single- and double-blind peer review.
- Open reports: in contrast to the traditional peer review where review reports are kept confidential, in the open form they are published with the relevant article.
- Open participation: The wider community and not just invited reviewers are able to contribute to the review process.
These modifications have been made in order to address the disadvantages of the classic peer review process, in particular its lack of transparency, lack of incentives, and wastefulness.
Understanding the mechanism: how the open peer review process works
The process of open peer review operates typically following certain steps. Upon submission, the author of the article usually signs a statement representing that they are the sole author, or in the case of co-authors that they have agreed to be represented; original work and research free from plagiarism has been conducted; the author has the right to deal with the intellectual property of the article; all experiments conducted during the research process have followed the relevant institution’s guidelines; the article has been written in compliance with the local, national and international laws, and it does not include any irrelevant or advertizing material. It should also be clearly stated that the author agrees for the article to be available online, as well as distributed and reproduced.
After the agreement has been signed, the article can enter the publication stage. It is most commonly made available online and reviewers are then invited. The author has the right to invite peers of their choice, but peers can also request permission to review the article. Each peer can only submit one review. All reviewers are also required to sign a statement agreeing that their identity and credentials will not be kept anonymous. They agree for their review to be publicly available under unrestricted use, distribution and reproduction. In addition, peer reviewers can sometimes be asked to provide a detailed account of possible financial or other conflicts of interest that could potentially bias either positively or negatively their review.
Following the signing of the agreement by the reviewers, the full text of the submitted reviews is published online under an open license which permits its free distribution and use. Reviews are posted and tagged to the original article according to the suitable technical standards. They are made open to commentary and evaluation. Authors may submit at any point an updated version of the article that cites relevant reviews, and which will also become available at the repository and be tagged to the original article.
It is academic journals’ top priority to ensure smooth running of the open peer review process. Especially because it comes as an opposition to the traditional peer review, its functional implementation is important and usually kept to.
More breakthrough alternatives
Some journals and publishers have been testing sub-methods of the open peer review which essentially encourage even more openness. For example, the transparent peer review enables open publication of an article’s entire peer review process in an easily accessible, searchable, and citable format in order to bring greater transparency to the research process and recognition to the work of peer reviewers.
The transferrable peer review allows subject-related journals to transfer reviewed manuscripts between each other. Typically, an author submits their paper to a journal but after it has been reviewed the editors decide that although not suitable for their journal it is likely to be appropriate for a similar journal. The author is then given an option to transfer the manuscript to the other journal. One of the main benefits of this model is immediately providing the author with an alternative outlet for their work and potentially speeding up the publication process.
The collaborative peer review covers a broad variety of approaches in which a team of people work together to undertake the review. One format is to have two or more reviewers work together to review the paper, discuss their opinions and submit a unified report. Another approach is to have one or more reviewers collaborate with the author to improve the paper, until it reaches a publishable standard.
However, the open peer review process remains the most preferred approach among the alternatives. It is especially so because the process has a number of benefits for all parties involved.
A breath of fresh air: the advantages of the open peer review
The open peer review process is defined first and foremost by transparency. This essentially means that peer reviewers who participate in the peer review process are not anonymous. Reviewers can be held accountable for their evaluations. This leads to a higher review (in terms of tone and quality) with fewer instances of reviewer bias as conflicts of interest can be identified by the participating community.
The open peer review process is characterized by complete openness: open identities, open reports, open participation, open interaction, open pre-review manuscripts, open final version commenting and open platforms. Generally, open peer review is open to anybody who wishes to participate. This means that there is reciprocal discussion between the participating parties, readers are also able to contribute to the review process, manuscripts are immediately available before the formal review, and reviews can be added to the final version of publications. It appears from various studies that open peer review is a valuable revision to the old peer review process and its anonymity.
Opening up what has traditionally been a closed process increases opportunities to spot errors, validate findings and to increase our overall trust in published outputs. Direct, reciprocal discussion between author and reviewers and/or between reviewers is not only allowed, but also encouraged. What is more, a lot of time is saved when using the open peer review form. Traditional peer review takes a long time. In many cases, it can be almost a year between submission and final publication, meaning that access to research findings are delayed substantially. Moving from individual peer reviewers to community review speeds up the process significantly by opening up the pool of reviewers able and willing to take on the review.
The open peer review process is undoubtedly more reliable than the classic procedure. Including the wider community instead of depending on just one or two reviewers provides a better opportunity to identify methodological flaws or other inconsistencies in the research. An open discussion enables constructive criticism and better understanding among peers and authors. Reviewers can have differing opinions about the papers they review. It can often be unclear as to why a paper is rejected by one reviewer and not by another. Open peer review supports improved consistency and reduces the chance of bias by ensuring that more reviewers' views for a given paper can be captured and compared.
Making reviewers' questions about a given paper open along with the responses from authors provides valuable context about the methodologies employed and research processes. This is only possible if review reports are made accessible. By doing that, not only the authors themselves benefit, but also the science community gains valuable information and insight.
A realistic judgment of open peer review
Open identities have been argued to incite reviewers to be "more tactful and constructive" than they would be if they could remain anonymous. Open peer review in all its forms has been argued to favor more honest reviewing, and to prevent reviewers from following their individual agendas. Some studies have found that open identities lead to an increase in the quality of reviews, while other studies find no significant effect.
It should be noted that the open peer review process is not a magical solution to the problems that the traditional process has. Open peer review is not short of its own disadvantages, although they are significantly less compared to its classic alternative. It has been concluded through various studies that open peer review does not dramatically affect the quality of review or the rate of acceptance of articles for publication. The only significant negative effect of open peer review has been proven to be an increase in the likelihood of reviewers declining to review. Open peer review is not perfect and does not mean that all bias will be eradicated or that senior researchers will not retaliate if younger researchers criticize their work.
That being said, it is more than likely to make such occurrences more open, which can lead in turn to reviewers' reputations being damaged when they act inappropriately. Emerging open peer review models are constantly evolving and improvements are being made as lessons are learned.