AGU Launches Hypothesis to Facilitate Peer Review

This is a cross-posting of an article by Brooks Hanson, Senior Vice President at the American Geophysical Union, originally at EOS, with added quotes.

We are pleased to announce that this month AGU is implementing a new way to review manuscripts across all our journals: Hypothesis. Hypothesis is a mission-driven organization working to build open-source software that enables annotation on digital documents. It is already being used in diverse fields including education, research, journalism and publishing.

In a traditional review, a reviewer might provide a long, itemized list of comments tied to pages and paragraphs or line numbers in a manuscript. Some also mark up a manuscript using track changes in Word or use the editing features in a PDF document while also providing other comments and a summary separately. Multiple reviews yield separate markups that cannot easily be merged.

Why not have all the comments in one document for the author, organized by who created them and tagged by the type and importance of a comment? Even better, also have the comments in list form that can be exported, filtered and sorted, and allow the authors to respond either in line, or using the list.

“The process of peer review can be improved. The easier and more efficient we can make it, the quicker we can move science forward. Annotation is a proven way of focusing observations and critique in the margin, right where it’s relevant,” said Dan Whaley, Hypothesis CEO. “This work highlights the potential utility of annotation from pre-publication through post-publication use cases and showcases the value of a consistent UX and workflow for creating annotations based on open source software and open web standards.”

We are implementing this solution for our journals by incorporating the Hypothesis annotation tool into GEMS, our editorial system. This work is a cooperative effort by eJournalPress, which provides GEMS, and Hypothesis, with initial support from the Alfred P. Sloan Foundation, and some extensive work and testing by the publications staff at AGU.

“eJournalPress has collaborated with Hypothesis and AGU over the past two years integrating the Hypothesis annotation tool to provide Editors and Reviewers with a enriched review process,” said Joel Plotkin, eJournalPress CEO. “We have presented this new integration at our annual US and UK user’s group meetings and it was well received. Our customers look forward to using this and other tools to advance the scientific process.”

The Hypothesis software allows each reviewer to make inline comments and organizes these by reviewers, editors, authors and importance all in one file. Confidential comments for the editor can also be included.

Currently, each reviewer will only see their annotations while the editor can see the full collection, and the authors can see all non-confidential annotations. Thus, anonymity and viewing rights are preserved.

Each annotation is labeled to help organize the review. Current labels include a summary comment, major issue, minor issue, text edit, comment on a figure, and comment regarding a reference, and these can be filtered for ease of viewing. The annotations can also be collated separately and formed into a “traditional” review.

Authors can respond inline to reviewers’ comments on the same file, providing a means for a more effective dialog among the participants in the peer review process.

Using annotations in a review does require online access (although annotations are saved immediately) but online availability has expanded greatly in the past few years. Some instructions are available here.

Conducting a review using annotations is entirely optional — a reviewer can still choose the traditional way if they wish. For those that do wish to try it, we welcome feedback. Feedback is important to refine and improve the experience and provide broader familiarity with this development.

We appreciate that any changes and innovations may take some time to get used to; however, this approach opens up some additional opportunities for collaboration and improvement in the peer review process.