In reading this post by Megan McArdle, it occurs to me that most people outside the sciences don’t understand what “peer review” actually is. They have a wildly exaggerated concept of how thorough and detailed the process is. My spouse pointed out that most lay people imagine that the experimenter presenting a paper for peer review is forced to cower before a bench of a half-dozen or more of his peers who then mercilessly grill the experimenter about every facet of his work.
In short, they imagine that peer review looks something like this:
…when in reality peer review looks like this:
By the way that proponents of Catastrophic Anthropogenic Global Warming (CAGW) wave it about as a talisman to ward off criticism, a lay person could be excused for thinking that peer review is a rigorous process that is central to the functioning of science and that verifies the conclusions of a scientist’s research.
Peer review is nothing like that.
Peer review isn’t even central to science. Science functioned fine for centuries without peer review and scientists who work in secret or proprietary environments do not use it. Instead, peer review serves economic and social functions related to scientific publishing and does nothing else. Peer review somewhat protects the integrity of scientific media, not the quality of science itself.
Peer review is a very superficial process more akin to a newspaper editor checking the grammar, spelling and punctuation in a letter to the editor before publishing it.
It works like this. An experimenter in a particular field sends a paper to a journal that covers that field. The editor then secretly selects scientists in the same field whom the editor believes are competent to glance over the paper and check it for obvious errors or faults. In the vast majority of cases, peer reviewers do not examine the original data, do not examine experimental records, do not examine the experiment’s hardware/software and they most certainly do not confirm the results claimed in the paper by reproducing the experiment themselves.
Saying a paper is peer reviewed says nothing about the validity of its conclusions.
Quite the reverse, it is the fate of most scientific papers to be proven completely wrong. For example, at present, there are dozens of different variation of string theory in physics, all of which contradict each other. So, we know for a fact that the vast majority of published peer-reviewed papers on string theory are dead wrong. We just don’t know which ones because we lack the experimental technology to test which string theory papers accurately predict the outcome of experiments. That doesn’t stop physicists from churning out papers on string theory and it doesn’t stop journals from publishing them. They do this to foster the scientific dialog, not because they have any idea which of the papers, if any, will eventually be proven correct.
Why then do we use peer review? Simple, publishing glaringly flawed papers or being seen as taking sides in a scientific dispute destroys journals both professionally and economically. Peer review protects the journals and the careers of the people who staff them.
A journal’s reputation is its fortune. That reputation is dependent on the journal’s publishing of papers that other scientists use for the basis of their own work. Scientists buy and read journals with good reputations and ignore those with poor reputations. Nothing so damages a journal’s reputation as does publishing a glaringly flawed paper and wasting the time of the scientists who base their own work on the flawed paper.
Peer review protects a journal’s reputation by hiring experts in a field to check papers prior to publication. It is not a journal’s responsibility to confirm or refute experimental conclusions, but it is their responsibility to check for basic errors in math or methodology, just as they would check for errors in grammar or spelling. Peer review offloads any responsibility for publishing bad papers onto anonymous members of the scientific community. It’s a perfect form of blame passing that everyone else wishes they could use.
This blame passing also keeps journals and editors from being accused of taking sides in personal and professional quarrels. It is also the reason that reviewers themselves prefer to remain anonymous. No scientist wants to suffer the professional and personal consequences from either refusing or accepting a paper they should not have refused or accepted. It is also why peer review is a superficial review. The reviewers do not wish to be dragged into the minutia of scientific debates and quarrels. Instead, they concentrate on the basics that everyone can agree on.
In the end, it is the anonymous and secret nature of the peer review process that marks it as not part of actual science. The entire point of science is that all observers of a phenomenon can agree they see the same thing. Critical to creating that agreement is ruthless transparency. Secrecy is antithetical to the functioning of science, and peer review is a secret process. Science is not settled by the secret complaints of the anonymous.
Mere peer review should never be the basis of public policy, because when you get right down to it, we have only the word of the journal’s editor that a peer review was even performed. There is no formal mechanism to assure that peer reviewed is performed or that the reviewers have the competence to review the paper in question. If the journal’s editor is corrupt, then there is no independent mechanism that forces a peer review or ensures its quality. The entire system is based on a presumption of trust and on the discipline of the free market in scientific publishing. If an editor wanted to fake a peer review, no one could tell for a very long time.
Even if everyone is honest, the inevitable professional biases of peer reviewers can cause them to reject papers that call into question the tenets upon which the reviewer’s own work rests. If a scientific field is relatively small and all the peers share the same scientific blind spots or misapprehensions, then peer review can’t catch even gross errors that become obvious in hindsight. It is common for peer reviewers to repeatedly reject papers that substantially alter a major tenet of a field. Most of the game-changing papers of the last century were rejected by multiple peer reviewers at multiple journals.
People who try to defend a scientific assertion by claiming it appears in a peer reviewed journal are making the weakest defense possible for the assertion. All it means is that some editor and the reviewers he selected thought it met their minimum quality standards for publishing. Once you raise the specter of political corruption on the part of editors and peer reviewers, it doesn’t even mean that.
Replication and proven predictive power, not the opinions of peers, test science assertions. Those iron objective tests separate science from all other disciplines. In the long run, scientific peers are always wrong. Science does not advance by consensus, it advances by the repeated destruction of each generation’s existing consensus.
If the strongest defense someone can muster for a paper’s conclusions is that “it has been peer reviewed,” that is a dead giveaway that the paper itself is very, very weak. Basing policy on scientific studies that have been merely peer reviewed is just flat insane.