Peer Review

One of the functions of the peer-review process is to identify, and if possible correct, problematic articles at the pre-publication stage. It is not only readers of peer-reviewed publications who are the beneficiaries of this process. Authors also benefit greatly from constructive criticism by peers.

Still, from the standpoint of those who participate in this process, the main function of the peer-review process is to act as arbiter deciding the merit, indeed the validity, of contributions.

Thus, consider for instance the following extract from an editorial published in Decision Point, a monthly magazine of the The Applied Environmental Decision Analysis (AEDA) hub, that publishes articles, views and ideas on environmental decision-making, biodiversity, conservation planning and monitoring (Possingham, Decision Point, issue 45, p. 2, 2010):

The Federal government took a calculated risk investing in a multidisciplinary centre that was very different from traditional ecological science.

And what has been the return on that investment? Quite a lot if you consider our achievements (many of which have been presented in Decision Point, see the next page for just a few examples). It’s important to note that all of these outputs appeared in the peer-reviewed literature (including some of the top journals like Science and Nature). We often forget that the CERF program is a research program, albeit applied research, and research must eventually be subject to peer review to be credible.

Now, although this position may well be held by most academics, this short paragraph glosses over a well-known fact that is rarely discussed in peer-reviewed journals. The publication of an article in a peer-reviewed journal — even a “top” journal — is no assurance that the analysis/theory presented in the paper is indeed “correct”. In fact, one of the weaknesses of the peer-review process is that, occasionally even badly flawed contribution can pass muster in the peer-reviewed process.

Of course one may counter that experience has shown that on the whole this process works well. And if mishaps occur, some of which may even be notorious (for example, see this famous case) these would be the exception rather than the rule.

The fact remains however that, because “bad apples” do pass the peer-review process the implication is that this process can be instrumental in endorsing flawed ideas/theories ex. Voodoo theories. And the trouble is that once this occurs, it can prove extremely difficult to convince followers of such ideas/theories (voodoo theories) of the unwarranted merit attributed to them in the peer-reviewed literature.

Unfortunately, a frank discussion of this issue is no easy matter. See the presentation The Power of the (Peer-Reviewed) Word for more …