Human Reviewers: The Achilles Heel of Scientific Journals in a Digital Era

Floyd E Bloom
(fbloom@SCRIPPS.EDU)

Virtual Keynote address:
5th Interent World Congress for Biomedical Sciences

Abstract
The Review of Scientific Manuscripts
The Review of Scientific Manuscripts Can Be Wholly Electronic
Why Do Authors Select Certain Journals for Their Best Papers?
Reviewers Are the Achilles Heel of the Review Process.
Conclusion: To Improve Quality and Speed of Review Requires Rewards for Reviewers

Abstract

The World Wide Web and other forms of digital communication have lead some to predict the demise of printed journals. In my experience, scientific articles can be more efficiently reviewed and edited by digital document sharing. High quality print journals will remain the preferred scholarly venue for authors's best works, but only if the performance of human reviewers can be improved and appropriately rewarded.


The Review of Scientific Manuscripts

Participants in this Conference are among the most experienced of electronic browsers for scientific content. Yet unquestionably, almost all of them are still enmeshed in a process of scientific manuscript submittal to their favorite journals that has been the standard for more than 200 years. As shown in Figure 1, an author submits a manuscript to a journal by delivering it to that journal's editor. The editor, having decided through various means often introspectively and subjectively that the paper would be suitable for that journal's mission, then ponders several questions. Who is a peer of this author, in content area, in technique and when possible in experience? Which reviewers should be avoided for competitive conflicts and past disputes? How much of the submittal's data can the editor expect the reviewer to examine in detail? For example, should the statistical conclusions be re-calculated? Should all the raw data from which the observations to be reported were extracted be made available to the reviewers? In short, what does the editor need to know to decide whether to accept or reject a submittal. The following personal observations are based on my experiences as the editor or reviewer for several scientific journals, including service for the past 4 years as Editor-in-Chief at SCIENCE.

[Reviewing process]

Figure 1: The standard procedure for manuscript submittal and review. An author selects a journal (1) by transmitting a submittal to that journal's editor. The Editor selects reviewers (2) who return their criticisms and suggestions to the editor, who then reinforces their views in correspondence to the author (3). Eventually, after one or more such rounds of review and revision, the editor will reject or accept the paper. The accepted paper is then transmitted to the publisher (4), who then includes the paper in the journal (5) as the production process allows, pending copy editing, return of author galley proof corrections, and barring complications in graphic reproduction.

[Back to the top.]


The Review of Scientific Manuscripts Can Be Wholly Electronic

Until quite recently, each of the steps of the scientific manuscript review process--submittal, review, revision, acceptance and eventual transmittal to the publisher-- required physically transporting documents, tables and figures from one player to another (see figure 1). For the past year, Warren Young and I have been conducting an experiment with Elsevier Scientific Publishing in which this entire process is conducted electronically through the World Wide Web without any hard copied materials, and without physical manuscript required. The SMART process (System for Manuscript Review and Authoring Technology) and the required authors' transmittal module are available on the Web at the URL http://smart.scripps.edu:8000/smart.html. The editor's choice of peer reviewers is aided by searching a database of qualified and self-declared reviewers on the basis of keywords in their areas of content and technical expertise Accepted papers are eventually transmit ted to the publisher, posted on their journal Web site within a few days, and eventually printed in the appropriate section of the journal Brain Research. Other systems for electronic submittal and review are already in existence in Astrophysics. The journal Pediatrics, and recently, the Journal of Neuroscience have committed to digital only 'publication' of papers submitted conventionally to their review process. Clearly, the review and the dissemination channels of scientific publishing can profit from digital document sharing.

[Back to the top.]


Why Do Authors Select Certain Journals for Their Best Papers?

The question of technological efficiency in the review process is but one element of the multi-faceted selection process by which authors decide the journal on which they wish to bestow their latest manuscript. As shown in Figure 2, authors select their journals on the basis of the past content of the journal in regard to a variety of factors in peer appreciation, timeliness of the manuscript selection and decision process, and the overall quality of the appearance , publication, and delivery of the journal. The reputation of the journal for its previous quality of papers and content is critical. How often does it appear? Who else has published there? How many subscribers does the journal have? Is it listed in major indices of publication? How many libraries carry this journal? Other aspects of peer appreciation include the speed with which its submitted papers are accepted for eventual publication, and the quality of appearance of the final manuscripts and artwork. In short, authors select journals for thei r works for the impact these publications will have on their scientific reputation for academic promotion and for support of their grant applications.

[Journal selection]

Figure 2: Factors in journal selection by authors. Authors select journals in part by bearing in mind several factors as noted in the text.


[Back to the top.]


Reviewers Are the Achilles Heel of the Review Process.

Almost all aspects of the scientific manuscript and review process can be accelerated by digital transmittal of manuscripts, illustrations, reviews, and revisions between authors, editors and reviewers. However, in order to maintain the apparent quality of their journal, editors seek the most wise reviewers on the basis of their past performance and for their scientific reputations as scientists. Often reviewers in 'hot' areas of science may be in great demand for reviews by many different journals. Reviewers working in relatively quiet research fields may also be in equally intense demand because the pool of qualified reviewers is quite small. Regardless of the reasons, only the most dedicated academic scholars undertake to review. There are many possible reasons for their participation in the process, but none have been studied scientifically.

From my personal perspective, some factors are suggestive: past pleasure in the scientific publication process, respect for the scientific review system, and anticipation of equally rigorous and fair reviews in the future. Often the 'best' reviewers are under their own pressures, such as research, grants and their own papers. No matter how quickly manuscripts and reviews may be digitally exchanged between editors and reviewers, the major temporal sink remains the delays in return of useful reviews from those asked to review. Journals with lesser reputations may suffer the most, since the needed reviewers will owe little allegiance to the journal, and the quality of their reviews may reflect their judgement of that journal's reputation. Delays in receipt of reviews with useful critical judgement may in turn delay the decision making, which in turn affects the journal's aggregate reputation. All in all, prompt, high quality reviewing is the essential element of the scientific manuscript review process, and the one most vulnerable to the vagaries of journal editor-reviewer interactions.

[Reviewers' role]

Figure 3:The critical role of reviewers. Editors select reviewers not only for their scientific expertise in certain areas of content and technology, but also for their past performance as reviewers in terms of open-mindedness to new ideas, including data which may overturn the reviewers' own findings. Editors also select reviewers for their capacity to respond with reviews to papers in a timely manner. While the authors, the publishers, and the editors are all compensated for their performances in this process, the reviewers in general serve only in anonymity, with the gratitude of all

[Back to the top.]



Conclusion: To Improve Quality and Speed of Review Requires Rewards for Reviewers

Given the importance of reviewers in the scientific manuscript review process, how can editors and their publishers offer incentives for exemplary performance. Monetarily, it is possible to provide free subscriptions to reviewers. That reward will have the most value when an expensive journal is worthy of the subscription charge (e.g., in terms of content quality, amount and timeliness of production). However, free subscriptions may be constrained by the goals of the publisher, whether they are commercial or not-for-profit. Dedicated scientists reviewing for their society owned journals may already be subscribers as part of their membership fee. Generally, the monetary value of a free subscription may well not reflect the time and effort required to review frequently. Editors can offer reviewers prestige, but appointing them to visible editorial positions, or inclusion in annual lists of reviewers, but this notification my feel to reach the members of appointment and promotions committees. One incentive rarely practiced is for editors to write to departmental chairmen to praise their faculty member's performance as a reviewer. Editors can offer reviewers some insight into their decision making process by providing them copies of the letters of transmittal from editor to author after the editor has read the reviewers comments.

By showing reviewers which kinds of assessment are most valuable in shaping a paper in need of revision, an editor can achieve several important goals. The editor can help educate reviewers to the quality standards of the journal and to the sorts of information that editors need to make their decisions to accept or reject. Lastly, it should be possible in the near term to offer potential reviewers, up-to-date and comprehensive lists of pertinent bibliographic materials related to the manuscript under review. Whether any of these possible rewards will be appreciated is also an experiment in progress. Nevertheless, the reviewer has been the most critical, and yet absolutely under-valued element of the scientific manuscript review process. Rewarding good reviewers for excellent performance is essential for maintaining the quality of the review process, no matter how quickly manuscripts are moved from authors to editors to reviewers.

[Back to the top.]


Bloom, Floyd; (1998). Human Reviewers: The Achilles Heel of Scientific Journals in a Digital Era . Presented at INABIS '98 - 5th Internet World Congress on Biomedical Sciences at McMaster University, Canada, Dec 7-16th. Keynote Address. Available at URL http://www.mcmaster.ca/inabis98/keynote/bloom/index.html.

© 1998 Author(s) Hold Copyright