Peer review…

Anyone involved in a research field that involves publishing your findings should probably be familiar with the concept of ‘peer review’. If not, you might be asking what peer review is, exactly? To quote the Virginia Tech University Libraries webpage, peer review is:

“The process of obtaining impartial opinions from the research and academic community in order to ascertain whether papers submitted for publication in journals or at conferences are of a suitable standard. The opinions are sought by publishers and conference organizers, and are requested from those whose expertise and stature are similar to the authors.”

The idea is simply that researchers provide quality control for the publication of scientific findings by their peers. While anyone could serve as a reviewer, in theory, the idea is that your research should be reviewed by professionals from your field, who are familiar with the type of work you are doing and best qualified to judge the merits or flaws in your research design and results. There are, of course, drawbacks to this approach. Politics and rivalries can come into play, as researchers might seek to delay the work of a rival, promote the work of a friend or worse, outright steal research ideas while serving as a reviewer. This is a potentially huge discussion that I am not going to delve into right now, but suffice it to say that the peer review system is far from perfect. Getting back to the topic at hand, what prompted this was a recent article that was written by John Bohannon and published by Science:

This article describes what the author refers to as a “sting operation” to “reveal the contours of an emerging Wild West in academic publishing.” The target of this ‘sting’ is the rapidly-growing number of open-access scientific journals that are available today. Again, to offer a little background for those not familiar with the process, the historical model for scientific publications is the subscription journal. This means that authors submit their article to a journal for review and (hopefully) publication, and the journal is then funded through subscriptions from people or institutions that want access to the information. Before digital information became so prevalent, this meant a subscription to the actual journal, much like you would subscribe to a magazine. Each week, month, quarter, or whatever you would receive your magazine in the mail. Maybe your institution would subscribe to it, and you could visit the library to read it. Today, with so much ready access to information via the internet, you might choose to pay for a regular subscription to download articles online, or just pay per article. As before, your institution might elect to pay for subscription fees to allow faculty, staff or students to download articles. The problem is that this places a cost restriction on the availability of information. In order for the scientific research community as a whole to function effectively, information should be shared between researchers who contribute to the ever-growing body of knowledge. We are all standing on the shoulders of giants, building upon the foundation provided by those who went before us. If you are working on a tight budget, or located in a developing country, you might have a difficult time gaining access to the information due to cost, legal restrictions, or just internet access issues. Hence the push for open-access journals.

Open access means that anyone, anywhere, should be able to freely access the information. There are different approaches, but in general, many charge the author a fee to publish, or find other ways to subsidize the operational costs. The Public Library of Science (PLOS) is one of the better-known organizations, operating quite successfully as a non-profit publisher and advocacy group. Much like subscription-access journals, the quality of the publications that are published is dependent upon the editors and reviewers. The difference between the two models, many will argue, is that subscription-access journals have an additional layer of scrutiny in the form of subscribers. Subscribers are less likely to continue paying for access to a journal that publishes low-quality research, and are even less likely to pay for a journal that published research with flawed designs, poor execution, faulty conclusions, etc. In an open-access model, since the fees are often paid by the author, there exists the possibility that less scrupulous owners or editors might yield to the temptation to lower standards in order to accept more submissions, and thus, more publication fees. Indeed, there has been an explosion in the number of open-access journals around the world over the last decade. With that explosion comes a growing suspicion that many of these journals are have lower standards, or worse, engage in practices that are unethical, or even designed to prey upon the unwary.

Now we come back to the original topic of the post, which was the recent article put forth by Science regarding the perceived lack of scrutiny on the part of many open-access journals. The author of the article fabricated a ruse to test the standards of a group of open-access journals. A false research paper was fabricated with intentionally inaccurate methods, flawed conclusions and exaggerated claims. This paper was submitted in slightly differing forms under various names to 304 different journals selected by author, many of which are on a list of publishers that are believed to be ‘predatory’. If asked for revision, the author made cosmetic revisions and resubmitted, but the goal was to see how many would ultimately accept the flawed paper and how many would reject it. Of the 255 journals that offered either and acceptance or rejection, 157 accepted the paper for publication, at which point the author withdrew it. This certainly represents a rather alarming lack of quality control, but what does it actually mean?

In general, I like the premise very much: testing the quality control of journals to demonstrate how successful they are at filtering out the bad science and keeping the good. What I do not like is the implication being made that this is necessarily indicative of a problem with open-access publication. This ‘sting operation’ was not actually set up as a proper, controlled study. The targets were hand-selected, and no subscription-access journals were targeted, so it is not possible to compare the two models based upon the results obtained here. For all we know, this could be equally prevalent among subscription-access journals as well. I find it disappointing that this would be used as a condemnation of the entire model of open-access publication, as Science seems inclined to do. I am not even sure that it can be viewed as an indictment of the peer-review system, as some people have suggested (Michael Eisen, co-founder of PLOS being one such person). The Open Access Scholarly Publishers Association (OASPA) response to this article, which I think did a good job of pointing out some of the major issues with the report, can be found here:

What it ultimately shows, as far as I am concerned, is that there are plenty of low-quality journals out there, and that researchers should be careful to identify the source and remain wary about reliability of information coming from such publishers. Unfortunately, there are probably always going to be people who seek to take advantage of others. It is incumbent upon the scientific community as a whole to identify and avoid such publications in any capacity. It is incumbent upon those who serve as reviewers to be reasonable and ethical judges of the work that they are asked to review. Finally, it is incumbent upon the publishing community to maintain high standards for the quality of work published, to promote the ethical behavior of reviewers, and to help the scientific community to identify publications that conform to these ideals. To paraphrase Bohannon, the Wild West is certainly no place for academic publishing.

Category(s): Science Stuff

One Response to Peer review…

  1. Because of so many different outlets and sheer freedom of speech, it will never be possible to prevent people from publishing bunk and claiming it’s great stuff.

    The reputation system is the only refuge. It simply doesn’t matter what a journal’s procedures are claimed to be. I also don’t think it much matters who is getting paid, or who is doing the paying. What matters is the track record of the journal. In my field, there are a bunch of open access journals nobody pays attention to because we know they’re paper mills.

    Which makes this a delightfully meta problem. See, we need peer review to keep the authors honest. We need community/peer review of the journals too, because it’s just plain naive to think that they’re serving science’s best interests. They’re just a bunch of humans running an organization that need to be able to pay the light bill. And like any other bunch of humans who need to pay the light bill, they’re not super-rational, and they’re not incorruptible.

    It’s a pain, and it takes time, and it’s a lot of work but we have to apply the scientific method everywhere, including to the scientific method. Don’t trust. Verify.