The Problem with Peer Review. A Standard that Doesn’t Always Translate Well in the Media or Courtroom
July 12, 2016
By: M. Shane Harvey
Most reputable scientific journals require that articles be “peer reviewed” before publication. Under this process, articles are submitted to a journal and anonymous “peers” (who have expertise in the subject matter) review the articles. Articles are then accepted, rejected or revised to address the comments of the peer reviewers.
The process is sensible. Certainly, scientific theories should be vetted in some way before publication. But the process has its limitations. And, unfortunately, these limitations are often poorly understood in the legal system and by the media.
For starters, “peer reviewed” does not mean “fact checked” or “replicated.” Many peer reviewers look at scores of articles every year. Their focus is on whether the author’s methodology makes sense. But reviewers do not replicate the author’s work or check the author’s math. In fact, the peer reviewer often does not even have access to the underlying data used by the author. The review process is closer to a “sanity check” than anything else.
As a result, work that is peer reviewed and published is often just wrong. A report by The Economist (“Trouble at the Lab,” October 2013) explored this issue in depth. In one infamous example highlighted by The Economist, a Harvard biologist purposely submitted a paper full of errors and design flaws to 304 journals. The result? The flawed research was accepted by 157 of the publications.
To compound all this, journals have a preference for publishing articles that find interesting or surprising correlations. A study finding “no link” between two variables doesn’t move the needle. But, an article proclaiming a link between cell phones and cancer, for instance, gets everyone’s attention. Unfortunately, as The Economist points out, the risks of getting “false positives” – that is, finding relationships that are not in fact true – can be quite high. In other words, the interesting stories are the ones most likely to be wrong.
None of this means that the peer review process is wrong. Sure, it could be stronger. Many journals have admitted that. But the scientific process works by having scientists replicate or disprove the work of other scientists. If bad science gets past the peer review process – and it will – the bad science should get weeded out in due time.
The problem arises in the interim. Courts and the media often treat peer reviewed studies as proven science, which they are not. Even when authors caution readers about the limitations in their study or the need for more research (and this happens frequently) the media often fail to highlight this. In terms of headlines, “Cell Phones Linked to Cancer!” is much more eye-grabbing than “Scientists Find Statistical Correlation Between Cell Phones And Cancer But Say Further Study Needed.” Likewise, the legal system tends to treat peer reviewed studies – whose authors are not subject to cross examination – as trustworthy, if not definitive.
If you find yourself on the blunt end of a “peer reviewed study,” you may experience a feeling of helplessness. Getting the media to understand and report the limitations of a peer reviewed study can be difficult. Likewise, taking on peer reviewed science in the courtroom can be hard as well. This is particularly true in obscure or novel areas of science, where a small number of studies can become “authoritative” simply because they are the only game in town. Nevertheless, it is important to understand the limitations of the peer review process and vigorously point these out in the media and courtroom. Good science will prevail in the long term. Your job is to make sure the scientific process has a chance to work. (This post may contain errors, but it has been peer reviewed.)
The article was authored by M. Shane Harvey, Jackson Kelly, PLLC.