Doctors with medieval weapons to go through to get paper published

Peer Review of Medical and Scientific Articles: Who decides what gets published, How do they do it, and why getting it right avoids fake news.

| 16 September 2019

The blog this week is published during Peer Review Week, an international event celebrating the essential role that peer review plays in maintaining scientific quality.

I invited Dr. David Maslove to co-author this blog, based on a Medical Grand Rounds (MGR) presentation he made that was voted by his peers to be the best MGR of 2018-19.

Dr. David Maslove is an Associate Professor and SEAMO Clinician Scientist in our Division of General Internal Medicine and the Department of Critical Care. His research focuses on informatics, genomics, and the use of biomedical Big Data to address current challenges in Critical Care Medicine.He is the inaugural Associate Editor for Data Science for the journal, Critical Care Medicine.

Introductory thoughts on peer review from Dr. Archer: Biomedical research passes through a process called “peer review” prior to publication. The concept of peer review is that a discovery is improved and validated by vetting by peers. Peers are experts in the field that the submitted paper deals with, and they are meant to be at arms-length from the author. Peer reviewers should be informed, interested and unbiased. Peer review in its purest form is based on scientific criteria. Reviewers consider the submitted paper based on the following questions: Are the methods and findings valid? Is the discovery novel? Will publication of the results advance knowledge? Peer review is not meant to be slow or censor transmission/translation of new knowledge to the public based on the personal biases of reviewers or the social mores of the day. Politics, gender, sex, race, ethnicity and age should not interfere with the evaluation of an article’s suitability for publication.

Peer review usually involves 3 or more peers anonymously rating the manuscript based on its novelty, impact and likely veracity (i.e. quality of design, robustness of the findings, clarity and quality of the Methods, compliance with ethical standards and adherence to appropriate statistical evaluation). The reviewers’ written comments are conveyed, through a journal editor, to the author. The output of peer review is usually several pages of written recommendations to the authors; however, peer review can be pithy and verbal, as in the following historical example. Nobel Laureate in Physics, Niels Bohr offered this peer review of Wolfgang Pauli’s presentation on Heisenberg’s field theory of elementary particles, “We are all agreed that your theory is crazy. The question that divides us is whether it is crazy enough to have a chance of being correct.” Ouch!

It is rare for any article submitted to a journal that uses true peer review to be accepted upon first submission (<5% of time). Virtually all manuscripts are identified as having flaws that can be improved through revision. Revisions (the bane of all authors) may range from minor text changes to major revision, requiring new experimental data and months of work. The editor and/or editorial board meet regularly, discuss all reviewed manuscripts (often hundreds per month) and pick from amongst them those judged to be in the top 10-20% by peers. These lucky few articles then move forward for revision, resubmission and hopefully eventual publication. The unhappy majority are rejected and their authors seek other journals and try again.

Peer review in biomedicine is most often conducted anonymously! This longstanding tradition is meant to encourage honest, fair assessments and allows reviewers to freely express their evaluation of quality free from fear of recrimination. Cloaked in anonymity they can encourage the author to improve the study and are well positioned to detect error, bias or fraud. This process gives the eventual consumer of the published information greater confidence in the veracity of the research findings. Anonymity can however lead to pettiness and mask hidden agendas. Prestigious journals demand a lot from authors during peer review and the when the process becomes petty, biased or unreasonable (asking for revisions that are so large they would themselves constitute a paper) authors may understandably cry foul and look for other publication venues (hence the rise of predatory journals).

Having been the beneficiary of peer review since 1983 I can offer my personal experience. Most of my articles have been substantially improved in the course of performing additional experiments or clarifying text to address the concerns of peer reviewers. On average my articles pass through 1-2 revisions. I have only had 1-2 papers (out of over 260) accepted without revision. In the past 5 years the number of revisions required is mounting. Peer review in 2019 is also often marked (in my opinion) by conflict of interest and harshness which was not meant to be the spirit of the process.

Whilst reviewers’ comments are sometimes pithy and to the point, other times they are more personal and less helpful. Moreover, peer review can be grindingly negative as there is a tendency to prefer comfortable, old ideas over disruptive, novel findings. Some reviewers are “live and let live” folks and will accept a manuscript for publication after a reasonable set of suggestions has been addressed (on the principle that it is the authors’ name, not their own, on the article). Other reviewers have major conflicts of interest; they know you too well and like you too little! Others are pursuing the same discovery and can’t help giving you suggestions that will slow your progress or have your paper rejected or bumped to a lesser journal. Some reviewers are just nasty-accusing authors of being misleading, naïve and worse…all in the guise of peer review. Indeed, some of the nastiest comments I have received relate to papers that once published became the most impactful and garnered the most citations. The harshness that authors perceive in the peer review process is shown in the cartoon below.

gauntlet of doctors with medieval weapons to go through to get paper published

Prestigious journals pride themselves on rejecting most papers and doing so rapidly. For example, Circulation Research, an excellent cardiovascular journal, advertises a 90% rejection rate. Nature goes one better. This high impact science magazine rejects ~93% of submitted papers and doesn’t use an editorial board, noting “Nature does not employ an editorial board of senior scientists, nor is it affiliated to a scientific society or institution, thus its decisions are independent, unbiased by scientific or national prejudices of particular individuals.”

The harshness and nastiness of peer review has led to gallows humor. I recently sent a paper reporting the discovery of a new gene that causes pulmonary hypertension to the Lancet. It sat there from March 8^thto April 17th and was neither rejected nor passed externally for peer review! Radio silence! Ultimately, when we persisted in asking the fate of our paper, we were informed that the paper could be passed on to a lesser, sister journal. It is true that journals have no obligation to send papers out for peer review and the Editor is entitled to single-handedly make the determination for acceptance/rejection of a submitted manuscript. Nonetheless, high handed behaviour has spawned humorous competitors, such as the Journal of Universal Rejection. This journal promises swift and immediate rejection, advertising the following points relevant to its virtues:

The benefits of submitting your paper to the Journal of Universal Rejection.

Likewise, the Journal of Irreproducible Results has as its mission statement “JIR appeals to scientists, doctors, science teachers, and word-lovers. JIR targets hypocrisy, arrogance, and ostentatious sesquipedalian circumlocution. We're a friendly escape from the harsh and the hassle. JIR makes you feel good :-)”. Peer review rarely makes you “feel good”.

I also deliver peer review as an Associate Editor or editorial board member for a number of journals and granting agencies in Canada and abroad. I try not to propagate the mean-spirited review practices that I have at times experienced. However, my observation is that there is a bit of a self-perpetuating cycle of abuse in which the recipient of harsh reviews becomes overly harsh reviewers themselves.

Several recent trends are changing peer review. One change for the good relates to anonymity. Some journals are eliminating anonymity, making the process of peer review part of the publication. They reveal the names and comments of the reviewers in the published article, thus providing the public with access to their critique of the work as it evolved through the peer review process. For example, I recently wrote an article on pulmonary hypertension for the British Medical Journal and one can see the reviewers and their comments on line. This is meant to reduce bias by reviewers and editors and also provides the reader with a sense of the enthusiasm/support/potential bias of the reviewers’ positions.

Other changes are less positive and promote false news in science, as discussed by Dr. Maslove. We live in the social media era where self-publication (on Facebook or through blogs like this) is the norm. It's fast and the acceptance rate is 100%! Perhaps it is the combined challenges of peer review and this expectation for immediacy that has fostered the rise of so called, predatory journals. Predatory journals are those that will publish anything one submits for a price without true peer review. While peer review can be slow and has flaws, predatory journals will rapidly publish anything, creating a vehicle for authors to promulgate beliefs that may be self-serving, flawed, spurious or fabricated. These journals often adopt names that closely mimic names of authentic journals, offer rapid publication, and publish at low costs. That reminds me to tell you that even in valid scientific journals you have to pay them page charges(often $5000 or more) once your paper is accepted for the pleasure of seeing your article published! Predatory journals also have page charges, but they are lower. Price is not the only appeal of the predatory journal. Want to publish an article saying there is no global warming or that the moon is made of green cheese? No problem just write it down and send it in with the publication fee and voila! Your paper appears in some rag with an odd name, let’s say JACF, the Journal of Astrophysics and Celestial Fromage! Universal acceptance is seductive! Predatory journals pretend to use real peer review but in many cases the editorial board is nonexistent. A recent article summarized the top 10 characteristics of potentially predatory journals (see Table).

list of characters of predatory journals

From Shamseer et al. BMC Medicine (2017) 15:28

Predatory journals are listed on various websites, so you can avoid them without too much difficulty.

Let’s turn things over to Dr. Maslove to learn how peer review is evolving and why we must guard and improve this mechanism of evaluation.

Dr. David Maslove’s text

According to a recent Pew Research Center study, more Americans are concerned with the problem of fake news than the problems of climate change, racism, and terrorism¹. It’s easy to understand why. Fake news has muddied political discourse in the U.S. and elsewhere, often to devastating effect. In our post-truth society it’s worth asking: has fake news also infiltrated the world of medical research? Recent developments suggest that the spread of misinformation – unintentional or otherwise – may be taking root here as well.

A critical bulwark against this trend is the time-honored practice of peer review, in which arms-length experts critically appraise manuscripts submitted for publication. This generates constructive comments to strengthen the work that eventually makes its way to print, while filtering out that which is substandard or irrevocably flawed.

Peer review has by most accounts served the scientific community well. But threats to its role as a safeguard against scientific fake news are beginning to emerge that if left unchecked could force widespread re-evaluation of what constitutes ‘truth’ in scholarly publishing. The stakes are high across academia, but medical researchers may feel the pinch sooner – and more acutely – than others.

While the motivation for scholars to publish their work is not new – the adage “publish or perish” was coined as far back as the 1920’s – the system of incentives in which they work has evolved considerably. No longer interested in simply disseminating their findings in a timely manner, scientists and clinicians now compete for both research dollars and academic jobs that are increasingly scarce. Additional confusion has been sewn in the digital age, with the proliferation of journals and other online research outlets. Scientific publishing has also become a highly profitable business, introducing yet another layer of complexity.

Yet the practice of peer review has failed to keep pace, having changed very little since its inception centuries ago. As a result, peer review is under threat from a variety of influences ranging from benign neglect, to malicious interference. A number of related factors illustrate this trend.

First, the peer review process may be swayed by reviewers who have a vested interest in the outcome. Ideally, reviewers are disinterested third parties, with no conflicts of interest relevant to the process. But many journals solicit suggestions from authors when choosing reviewers, which if accepted introduce the possibility of significant bias. An author’s colleagues may wind up vetting the work and may stand to benefit in some way from a favorable review.

In its extreme, this practice can lead to outright fraud, in which authors nominate fake peer reviewers, submitting glowing recommendations on their behalf. These biases increase the risk that substandard work gets published under the imprimatur of peer reviewed science, an outcome that can result in the pursuit of false leads, or the widespread adoption of questionable clinical practices.

Bias can also have the opposite effect, impeding the publication of high-quality science. Some reviewers may unfairly or unjustly criticize the work of their rivals, while others still might hold the process hostage in an attempt to have more of their own work cited in the manuscript under consideration.

Second, peer review can be hobbled by reviewers that are hurried, careless, or lacking in relevant expertise. Proper peer review is a highly skilled task that requires time and effort on the part of motivated and conscientious reviewers. The challenges of having a small pool of suitable reviewers are compounded by a low rate of acceptance of invitations from editors to review submitted manuscripts. As unpaid work that all too often goes unrecognized, reviewers must be driven by altruism, a commitment to the greater good, and a willingness to sacrifice time spent on their own research, or seeing patients.

But it’s not just reviewers who are showing more reluctance to engage in the process of peer review. Authors are increasingly exasperated by what is perceived to be a time consuming, laborious process that is fraught with inconsistent results. The outcome of peer review often feels arbitrary or misguided, leading to frustration, nihilism, and research findings kept under wraps.

Today’s publishing landscape offers these frustrated scholars new opportunities to disseminate their work outside of conventional channels. Open access publishing, in which authors pay article processing charges to make their papers freely available, has fostered a sense of comfort with paying to publish. This sensibility has created the conditions for yet a third problem, in which this willingness to pay is exploited by so-called predatory publishers. These are groups that solicit submissions to journals that may appear legitimate, but that lack the editorial trappings of a rigorous scholarly publication; peer review at these journals may be superficial at best, and at worst completely nonexistent.

Lastly, medical researchers are lately choosing to bypass journals entirely, or at least get their work out before the peer review process has run its course. This is achieved through the use of preprint servers, online repositories that host electronic copies of manuscripts posted without any form of peer review whatsoever. In some cases, the authors submit their manuscripts to a peer reviewed journal as well, either simultaneously or shortly thereafter, but there is no requirement that they do so.

Preprint servers have been well received in the physics community, where repositories such as arXiv.org have been in operation for more than a decade. More recently, bioRxiv.org has provided a similar outlet for the biomedical sciences. And just this year, medRxiv was launched. Billed as a preprint server for medicine, medRxiv – a joint venture between Cold Spring Harbour Labs, Yale University, and the BMJ – will rapidly post articles about clinical research as well².

But the response has not been one of unanimous support, with some critics arguing that medicine is inherently different from physics, and that premature publication of unfiltered research findings could bring unintended consequences³. The growth of medical preprints has the potential to spread information that is incomplete, inaccurate, or improperly contextualized. Since patients are often highly motivated to seek out medical information online, medical preprints could endanger the public health.

Heeding these warnings, medRxiv will include a series of “light-touch” processes to safeguard their process. These include verification that the work is in fact a research article, that ethical clearance has been granted, and that conflicts of interest have been disclosed. They also include overview by domain experts who will review the content (without explicitly endorsing it), and make a determination about its potential to do harm to individual patients, or the broader public. These safeguards are in essence peer review by another name, and amount to a tacit admission that peer review is indispensable to medical research.

Peer review is far from perfect, but it’s the best system we have to ensure the integrity and veracity of medical research. While authors now have more ways than ever to sidestep peer review, we see another way forward: lets shore up the process to make it even better.

Research into peer review best practices has begun in earnest, but more could be done. Journals are beginning to experiment with new models, including open peer review, and double- or triple-blinding to mitigate the risk of bias. Engagement is also key. If more people got involved in peer review we could realize meaningful gains in speed, transparency, and equity that would go a long way to addressing current shortcomings.

But perhaps the biggest impediment to improving peer review is the lack of recognition it receives. Doctors and medical researchers are motivated by a desire to help others, and to expand our knowledge. But they’re also people, with real-world concerns about their careers and reputations. We need to create mechanisms to incentivize members of our community to get engaged in peer review.

The optimal method to incentivize this work is not yet clear. Financial compensation, whether paid directly to the reviewer or to their institution, is one obvious option. But this comes at the risk of commodifying peer review, which may invite unwanted consequences⁴. Different incentives should be formally explored and evaluated.

Many of these will be journal-driven, such as frequent reminders, and “thank you” notices. More novel mechanisms include publishing the names of peer reviewers alongside the articles they reviewed, and conferring credits that can be used to offset article processing charges for the reviewers’ own future submissions. Others may be as simple as educating potential reviewers on the personal benefits they stand to realize. Peer review is one of the best ways to get an inside glimpse at what others in your field are doing, and a preview of the very latest findings. It also helps to establish the reviewer as an expert in the field, and can be a powerful tool for networking.

Perhaps the most meaningful changes will involve fundamental shifts in our institutional practices. Peer review could be recognized for the essential contribution that it is, and accounted for in promotion and tenure decisions.

At our own institution, we encourage our faculty to shape tomorrow’s practice by engaging in peer review today; this is at the very core of our mandate as an academic health sciences center. We encourage reviewers to track their contributions using Publons, a website dedicated to capturing peer review activities. We further encourage faculty to accept invitations for peer review regularly, and to seek advice from colleagues and mentors on how to get invited to review for the top journals in their fields.

Lastly, and most significantly, we commit to revising our current practices and policies in order to meaningfully credit faculty for their work as peer reviewers. The line connecting research to the bedside runs straight through peer review. We therefore see peer review not as an optional add-on activity, but as central to our mission. We’re committed to adding our expertise, our reputation for excellence, and our voice to this process, in furtherance of our mandate as a world class academic health center. We hope our approach will serve as an example to other universities; real change will come with harmonization of practices across institutions.

For all its dangers and faults, the emergence of fake news has brought with it at least one concrete benefit; traditional news organizations – only recently under threat by online upstarts – have seen a resurgence in paid subscribers. Media consumers recognize the inherent value of journalistic best practices and editorial oversight in making sure the reporting they read is as “true” as possible. Let’s do the same for medical research and make peer review great again.

References:

1. Pew Research Center. Many Americans Say Made-Up News Is a Critical Problem That Needs To Be Fixed. https://www.journalism.org/2019/06/05/many-americans-say-made-up-news-i…. Published June 5, 2019. Accessed June 18, 2019.

2. Rawlinson C, Bloom T. New preprint server for medical research. BMJ. June 2019:l2301-l2302. doi:10.1136/bmj.l2301.

3. Maslove DM. Medical Preprints—A Debate Worth Having. JAMA. November 2017. doi:10.1001/jama.2017.17566.

4. Chetty R, Saez E, Sándor L. What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics. Journal of Economic Perspectives. 2014;28(3):169-188. doi:10.1257/jep.28.3.169.

Final words: The veracity and value of peer review (and the resulting publications in journals) is extremely topical as our alternate funding plan, The Southeastern Ontario Academic Medical Organization (SEAMO), considers how best to evaluate and reward research productivity at Queen’s University. We want to use metrics that give credit to real discovery and the creation and dissemination of verifiable and valuable knowledge, while avoiding being hoodwinked by disinformation or misinformation that is on the rise in predatory journals that are propagating the scientific equivalent of false news. The first step to avoiding the propagation of false news is making sure the peer review process is fair and transparent, thereby keeping authors in the mainstream publication system. I encourage you to participate in peer review. It is rewarding and educational (and a responsibility of academic faculty members). Please be fair, balanced and to adhere to the golden rule (treat the author the way you would like to be treated). An important second step to maintain our system of peer review is to recognize predatory journals exist, avoid using them, and avoid rewarding/acknowledging publication in these outlets.