How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

This post by Rusty Speidel was originally published on the Center for Open Science blog on 6 Aug 2018 and is reposted here with permission.

We believe more discourse around research is a good thing. To that end, we have partnered with Hypothesis, a third-party platform, to allow for annotation and discussion on our preprints services. Annotation on preprints will increase transparency in scientific practices by enabling researchers to collaborate, discuss research with peers, and share additional information directly on preprints both before and after they are posted.

The partnership builds on the foundations laid by the publication of web annotation as a standard by the W3C, the pioneering work by Hypothesis to build an open annotation infrastructure and by COS to offer OSF a comprehensive platform for open scientific research and publication.

To get a sense of how the preprint services we support are thinking about increasing scientific discourse through features like commenting, we connected with Ben Brown from PsyArxiv, Philip Cohen from SocArxiv, and Nick Wehner from MarXiv to get their thoughts.

What prompted you to add the public commenting to your services?

BEN: We are excited to see commenting features come to PsyArXiv! We envision PsyArXiv as a way to not only disseminate your work, but going forward, a place to receive informal peer review. This would give you an opportunity to strengthen your manuscripts before submission for publication, if you decide to pursue that. This feedback from colleagues would be open for all to see, increasing transparency in the research process. We see integration of Hypothesis commenting as another way to make our science stronger and more open.

PHILIP: We run SocArXiv so people can share work faster and more efficiently, to increase engagement and interaction with fellow researchers and interested readers. Relying on journals and conferences is too slow, and often impedes sharing. One reason social scientists like the peer review process – in addition to all the reasons they don’t like it – is that it affords the opportunity to get serious feedback from fellow scholars. We hope that a robust commenting system on SocArXiv will provide that benefit as well, along with increasing the speed and dynamism of the interactions.

NICK: We believe community peer review is an essential part of the open science publishing workflow, so adding Hypothesis commenting to MarXiv was a no-brainer!

Was the decision unanimous among your advisory boards, your did you get dissenters? If so, what were their concerns?

BEN: We did not have any dissent within the steering committee about commenting. There was some concern about the potential pitfalls of anonymous commenting, but use of OSF accounts for commenting mitigates these concerns.

PHILIP: We were unanimous. The only concern we have is moderation and potential abuse. We don’t expect a lot of problems, but even a few people can cause a lot of trouble on an open system. So that’s one thing we will be looking at.

NICK: I didn’t ask anyone — I just told our Board this feature was in the pipeline, and we were adding it once it became available, but no one objected!

What do you see as the major benefit to adding commenting to preprints?

BEN: I see the benefits as two-fold: Commenting will provide a way for researchers to solicit feedback about their work pre-publication. This can, in the long run, improve the quality of work in our field. Further, we like the idea of comments being available post-publication as well. Authors could get feedback on research ideas they have already published and get feedback on measures, methods, and analytic tools their lab uses. This might even serve to facilitate collaborations. Secondly, public comments increase transparency. Peer review has historically been an opaque process to researchers and laypeople alike. Making this feedback openly accessible to everyone can help increase the public’s understanding and trust of scientific work. Public comments allow people who volunteer their time to provide essential feedback to get credit for their efforts (and possibly even authorship if their comments lead to substantive changes to a manuscript!). It also enables direct, clear address and discourse with reviewers/commenters. Public commenting ensures everyone participating in a dialogue around an article is a known entity, not a shadowy figure behind several walls of institutional protection. It increases transparency in these ways, which benefits the work being produced as well as (often) the people involved, we think.

PHILIP: We hope people will see commenting on papers at SocArXiv as an integral part of their research process, and incorporate the feedback they get into revisions, and generate new ideas and collaborations. Even if many community members are reluctant to take these steps at first, we are delighted to have the opportunity to contribute to a system like this now, so that we can push forward on developing new norms and best practices.

NICK: Not everyone is comfortable emailing authors of papers their opinions. In addition, it’s rare for a paper to list the contact information to all contributing authors. With commenting like Hypothesis, it’s really easy and quick to share your thoughts, freely and openly, without having to worry about tracking down contact info for the author. And if someone’s already asked the same question you did, you can publicly see that question and response. Comments might even strike up new research collaborations, which otherwise wouldn’t happen if this was all taking place offline.

How do you think it will be used by your communities?

BEN: Our hope is that the community will embrace public commenting on preprints. We have seen the emergence of preprints lead to a good bit of discourse (typically happening on social media sites like twitter and facebook), but these comments are not housed with the preprint in question. We imagine that having this opportunity to provide feedback that is stored directly with the preprint will increase collaboration at all stages of the scientific process.

PHILIP: We don’t yet know how it will be used. Our long term goal is to develop innovative alternatives to the existing peer review system. We want faster, more efficient interaction, we want high quality feedback earlier in the research process, we want the dialogue to be part of the scholarly record.

NICK: The marine conservation community is very results-oriented. Folks want to know what works and what doesn’t in the field. I suspect conservationists will share anecdotal evidence for how research is being applied in the real world, so authors can know if their results hold up out in the water.

How do you plan to promote its use among your authors?

BEN: We plan to let people know about the new commenting functionality through the usual social media channels. In fact, we have considered putting together our own blog post about this.

PHILIP: Don’t know yet, those plans are still evolving!

NICK: The MarXiv Team does a lot of communications to make researchers aware of the service, and how supportive traditional for-profit publishers are about sharing research freely and openly. We plan on adding the Hypothesis commenting to our regular communications blasts (individual requests to authors asking to share their papers) as well as our newsletters, webinars, social media posts, etc. When we see research shared in MarXiv that we know members of our community are working on, we’ll also alert them about the new research and invite them to share their thoughts with Hypothesis.