As we near the end of the semester, I’ve been reflecting on my journal entries from previous weeks. Although at first the topics seem scattered and random, upon further inspection several common themes began to take shape. Over the next several days, I’m going to be releasing a series of blogs where I explore these common themes from my journals. For the first blog in the series, I’m going to talk about the peer review process and its role in the scientific community.
Peer Review – Necessary or Just Friction? In the Open Access Week keynote lecture by Cameron Neylon, he describes peer review as something that adds unnecessary “friction” to the dissemination of scientific knowledge, impeding the process. In this same presentation, Dr. Neylon talks about peer review as “outsourcing,” and, as I understand it, implies that we are paying other people to do our thinking for us (when it comes to selecting which papers are of better quality). I think this might be true, but I don’t see the problem with it – like most things, we pay for it because we don’t want to do it. Although I think a change could happen in smaller steps, with time, I think shifting the burden of applying the “filter” at the user end all at once would be overwhelming to researchers. I have heard arguments saying if the current peer review system was eliminated today, shifting the burden of determining quality to the user, the old publishing system would begin to reestablish itself. The argument was that is that in the vacuum caused by the absence of peer review, it is likely that websites would be created that compiled lists of “best of” articles in a certain discipline. Once these sites became established, some authors would begin sending their findings to those sites directly in order to draw attention to their latest research. In order to meet this demand for review, these sites would need money . . . and so on. Of course, there are counterarguments to consider (Wikipedia, for example), but it’s an interesting point that I would have liked to see addressed in more detail.
What Peer Review Can’t Do For Us. In the NOVA program “Do Scientists Cheat?” one scientist described the role of peer review as “throwing out garbage.” In other words, peer review is designed to eliminate those studies that are clearly of poor design or low quality. However, peer review has limitations. In particular, the peer review system, used properly, can be very effective at spotting incompetence, but it is not designed to catch fraud. In a team project I worked on for another class, we noted in science, in particular, a surprising amount of the current system is based on trust. We trust that the paper we are reading is using data that was obtained legitimately, that substantial amounts of data were not thrown out, and that the authors actually did the experiments they are reporting. When that trust is violated, as in high profile fraud cases like that of Jan Hendrik Schön, we are unprepared, and science as a whole suffers. It is possible that increased transparency of the scientific process, through either providing datasets with each publication or, more radically, through open notebook science, will decrease fraud, or at least make it more difficult to commit, but, at some level, I think that we as scientists need to accept that “trust is the foundation of science.”
I’ve talked a lot about my opinions on peer review, but I really want to hear from YOU, the reader. In particular:
- What do you think of peer review: necessary or just friction in the system?
- How do you feel about the idea that trust is the foundation of science?