Running parallel to discussions about toll access vs. open access in academic publishing is a debate about whether it’s time to replace the venerable practice of (mostly anonymous) peer review with something more open, democratic, and dialogical. Web-based publishing software makes this technologically feasible–easy, in fact–and a number of journals are experimenting with it. A fine example of a new journal whose selection process is based on open reviewing is the World Economics Journal. The WEJ‘s description of its reviewing process is a model of clarity. (A hat tip to Rachael Tackett, a student at USF-Tampa, for pointing me toward this example.) As in many such endeavors, though, the proof is in the pudding, and it is by no means clear that such systems are sustainable in practice, especially for journals that receive hundreds of submissions annually.
As we saw in the first discussion of this series, plenty of anthropologists question the wisdom of fiddling with a system of peer review that underlies recognized quality standards, which are of course linked to retention and promotion reviews at academic institutions. But there are other reasons why we might pause before jettisoning peer review in favor of some version of open- or crowd-sourced review. The crucial role of blind reviews in teaching young professionals how to write and argue more effectively has recently been considered by the historian Zachary Schrag in a blog piece worthy of consideration. If you’re in the anthro business long enough, you’re likely to be burned by wrong-headed peer review. But for every instance of negilence, I’m reminded of five in which reviewers and editors provided frank, critical and, above all, useful thoughts about how to strengthen a submission or find it a more appropriate home. How confident are we that such advice would be forthcoming in an open-review system?
Given the nature of the theme, I thought that it would be fun to crowd-source this post. What follows are three observations from members of CFPEP or, in Richard Handler’s case, someone who was recently involved in a CFPEP-sponsored webinar.
From Deborah Nichols, CFPEP chair: “Nature, among others, conducted an experiment in open review of articles but found that most submissions received no comments or only one. Such an open approach would need to overcome the Lake Woebegon effect with its illusory superiority and the chilling effect of possible recriminations, retaliations, and lawsuits, not unknown in the academy.”
From Oona Schmid, AAA Director of Publishing: “Under the current system (blinded closed pre-publication review) almost all authors who submit relevant work to a journal can anticipate an editorial letter summarizing the comments of several reviewers. However, herding scholars to submit comments can take a long time; and of course these reviewers are uncompensated and don’t get credit for their anonymous contributions. An open system might greatly accelerate the time of publication (possibly reducing as much as 12 months from the process) and of course credits reviewers. I think the tradeoff might be that a large number of worthy ideas might not receive any feedback. Nature experimented with open review and found that some 11% of the papers received 53% of the comments; 42% received one comment; and 47% received no comment.
From Richard Handler: “Anthropologists should know that in human life, based as it is in language and culture, there can be no such thing as transparent communication. Every message, every communication, is culturally and linguistically particular, and ‘thick,’ also. So the idea that jiggering with the rules of peer-review will lead automatically to greater transparency is misguided. In a peer-review process, allowing more people to comment, with gate-keeping envisioned in terms of mass society instead of specialized guilds, might lead to some interesting intellectual exchanges. But it will not yield ‘results’ that are more transparent, more rational, or fairer than the conventional processes by which we currently operate.”
None of these quibbles and objections necessarily preclude experimentation with post-publication crowd-sourcing, although it may take a fair amount of editorial oversight to referee such discussions–yet more work for overworked editors and their staffs (if they have them!).
What do you think?