• 2016 AA Editor Search
  • Get Ready for the Annual Meeting

    From t-shirts to journals, 2014 Annual Meeting Gear Shop Now
  • Open Anthropology
  • Latest AAA Podcast

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 20,597 other followers

Rethinking Peer Review

Running parallel to discussions about toll access vs. open access in academic publishing is a debate about whether it’s time to replace the venerable practice of (mostly anonymous) peer review with something more open, democratic, and dialogical. Web-based publishing software makes this technologically feasible–easy, in fact–and a number of journals are experimenting with it. A fine example of a new journal whose selection process is based on open reviewing is the World Economics Journal. The WEJ‘s description of its reviewing process is a model of clarity. (A hat tip to Rachael Tackett, a student at USF-Tampa, for pointing me toward this example.) As in many such endeavors, though, the proof is in the pudding, and it is by no means clear that such systems are sustainable in practice, especially for journals that receive hundreds of submissions annually.

As we saw in the first discussion of this series, plenty of anthropologists question the wisdom of fiddling with a system of peer review that underlies recognized quality standards, which are of course linked to retention and promotion reviews at academic institutions. But there are other reasons why we might pause before jettisoning peer review in favor of some version of open- or crowd-sourced review. The crucial role of blind reviews in teaching young professionals how to write and argue more effectively has recently been considered by the historian Zachary Schrag in a blog piece worthy of consideration. If you’re in the anthro business long enough, you’re likely to be burned by wrong-headed peer review. But for every instance of negilence, I’m reminded of five in which reviewers and editors provided frank, critical and, above all, useful thoughts about how to strengthen a submission or find it a more appropriate home. How confident are we that such advice would be forthcoming in an open-review system?

Given the nature of the theme, I thought that it would be fun to crowd-source this post. What follows are three observations from members of CFPEP or, in Richard Handler’s case, someone who was recently involved in a CFPEP-sponsored webinar.

From Deborah Nichols, CFPEP chair:  “Nature, among others, conducted an experiment in open review of articles but found that most submissions received no comments or only one. Such an open approach would need to overcome the Lake Woebegon effect with its illusory superiority and the chilling effect of possible recriminations, retaliations, and lawsuits, not unknown in the academy.”

From Oona Schmid, AAA Director of Publishing: “Under the current system (blinded closed pre-publication review) almost all authors who submit relevant work to a journal can anticipate an editorial letter summarizing the comments of several reviewers. However, herding scholars to submit comments can take a long time; and of course these reviewers are uncompensated and don’t get credit for their anonymous contributions. An open system might greatly accelerate the time of publication (possibly reducing as much as 12 months from the process) and of course credits reviewers. I think the tradeoff might be that a large number of worthy ideas might not receive any feedback.  Nature experimented with open review and found that some 11% of the papers received 53% of the comments; 42% received one comment; and 47% received no comment.

From Richard Handler:  “Anthropologists should know that in human life, based as it is in language and culture, there can be no such thing as transparent communication.  Every message, every communication, is culturally and linguistically particular, and ‘thick,’ also.  So the idea that jiggering with the rules of peer-review will lead automatically to greater transparency is misguided.  In a peer-review process, allowing more people to comment, with gate-keeping envisioned in terms of mass society instead of specialized guilds, might lead to some interesting intellectual exchanges.  But it will not yield ‘results’ that are more transparent, more rational, or fairer than the conventional processes by which we currently operate.”

None of these quibbles and objections necessarily preclude experimentation with post-publication crowd-sourcing, although it may take a fair amount of editorial oversight to referee such discussions–yet more work for overworked editors and their staffs (if they have them!).

What do you think?

6 Responses

  1. Maybe editorial boards need to select papers that are most likely to get published. Then the editorial boards would need to heavily advertise the open comment period. It is very important that everyone is allowed to view the papers on the open web and not have to register using their name nor email address. It is very important that people would be able to leave comments anonymously if they wish.

    There could be a traditional peer-review process alongside a digital review process.

    Previously attempts at this open review system did not work at Nature, because there were too many papers. Apparently there were literally hundreds of papers and the Nature journal did not do a very good job at advertising. From what I heard, users had to register and the journal used a platform that was very difficult to operate.

    Papers for review also need be categorized so it is easy for professionals in certain fields to be able to look-up papers. There need to be opening and closing dates for comment periods. Most of all, the peer review system would need to be heavily advertised through traditional means such as Anthropology News, but also through social media (Facebook, Twitter, etc).

  2. […] rule! From this moment forward, in anything claiming to be a “discussion” of open peer review, no one is allowed to refer to the Nature experiment as evidence that open review can’t work, […]

    • LOL. I don’t think anyone could accuse Michael of stiffling debate, when I believe he has gone to great efforts to invite dialogue.

      At one point I played with comments along the following lines:
      The current system (blinded prepublication review) provides three things: 1) The process selects the best ideas; 2) it provides feedback to authors; and 3) it confers a stamp of approval.

      A digital-only publication can be on a less-restricted page diet –although online publishing does not eliminate costs (http://blog.aaanet.org/2011/10/25/more-on-aaa-publishing-the-matter-of-costs/) and so an online only title may not require peer-review for the purposes of content selection.

      Almost all relevant authors receive feedback from blind peer review, whether they go on to publish their revised papers in that journal or another. The Nature experiment suggests that openness _may_ cut both ways and we should all be aware that such a system may yield comments at high- and low-quality extremes, but may not provide all authors with constructive guidance that benefits many in the current system.

      If open peer review stifles feedback and a post-publication review journal by definition is less selective, publication in that journal will not legitimize scholarship to the dons of academe.

      Because I hear that the most important scholarly needs of our authors are feedback and academic jobs and promotion, any changes to our peer review system should focus above all on those two requirements. Possibly a model like the EGU’s (see: http://www.atmospheric-chemistry-and-physics.net/review/review_process_and_interactive_public_discussion.html) could accelerate publication times, increase debate with discussion papers, and ultimately select the best and thus still help credential our authors within the current academic system of evaluation.

  3. In addition to the resources and discussions already cited/linked to, readers of this exchange might find this paper, just published in PLoS One, of interest. It addresses some of the associated questions via experimental research. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0026895

  4. Another example for those considering this theme are projects like the edited volume _Writing History in the Digital Age_. http://writinghistory.trincoll.edu/ I just saw notice that today it passed the 500th comment point.

    While we tend to think of these efforts as new and technology dependent, they built on a older tradition of circulating pre-prints and working papers in route to the publication of a stable version.

  5. […] to the sessions to view these papers, to look at earlier blog posts (October 18, October 25, and November 9), and to join us in thinking about the long-term options and opportunities. Share […]

Comments are closed.


Get every new post delivered to your Inbox.

Join 20,597 other followers

%d bloggers like this: