In reading this post by Megan McArdle, it occurs to me that most people outside the sciences don’t understand what “peer review” actually is. They have a wildly exaggerated concept of how thorough and detailed the process is. My spouse pointed out that most lay people imagine that the experimenter presenting a paper for peer review is forced to cower before a bench of a half-dozen or more of his peers who then mercilessly grill the experimenter about every facet of his work.
In short, they imagine that peer review looks something like this:
…when in reality peer review looks like this:
By the way that proponents of Catastrophic Anthropogenic Global Warming (CAGW) wave it about as a talisman to ward off criticism, a lay person could be excused for thinking that peer review is a rigorous process that is central to the functioning of science and that verifies the conclusions of a scientist’s research.
Peer review is nothing like that.
Peer review isn’t even central to science. Science functioned fine for centuries without peer review and scientists who work in secret or proprietary environments do not use it. Instead, peer review serves economic and social functions related to scientific publishing and does nothing else. Peer review somewhat protects the integrity of scientific media, not the quality of science itself.
Peer review is a very superficial process more akin to a newspaper editor checking the grammar, spelling and punctuation in a letter to the editor before publishing it.
It works like this. An experimenter in a particular field sends a paper to a journal that covers that field. The editor then secretly selects scientists in the same field whom the editor believes are competent to glance over the paper and check it for obvious errors or faults. In the vast majority of cases, peer reviewers do not examine the original data, do not examine experimental records, do not examine the experiment’s hardware/software and they most certainly do not confirm the results claimed in the paper by reproducing the experiment themselves.
Saying a paper is peer reviewed says nothing about the validity of its conclusions.
Quite the reverse, it is the fate of most scientific papers to be proven completely wrong. For example, at present, there are dozens of different variation of string theory in physics, all of which contradict each other. So, we know for a fact that the vast majority of published peer-reviewed papers on string theory are dead wrong. We just don’t know which ones because we lack the experimental technology to test which string theory papers accurately predict the outcome of experiments. That doesn’t stop physicists from churning out papers on string theory and it doesn’t stop journals from publishing them. They do this to foster the scientific dialog, not because they have any idea which of the papers, if any, will eventually be proven correct.
Why then do we use peer review? Simple, publishing glaringly flawed papers or being seen as taking sides in a scientific dispute destroys journals both professionally and economically. Peer review protects the journals and the careers of the people who staff them.
A journal’s reputation is its fortune. That reputation is dependent on the journal’s publishing of papers that other scientists use for the basis of their own work. Scientists buy and read journals with good reputations and ignore those with poor reputations. Nothing so damages a journal’s reputation as does publishing a glaringly flawed paper and wasting the time of the scientists who base their own work on the flawed paper.
Peer review protects a journal’s reputation by hiring experts in a field to check papers prior to publication. It is not a journal’s responsibility to confirm or refute experimental conclusions, but it is their responsibility to check for basic errors in math or methodology, just as they would check for errors in grammar or spelling. Peer review offloads any responsibility for publishing bad papers onto anonymous members of the scientific community. It’s a perfect form of blame passing that everyone else wishes they could use.
This blame passing also keeps journals and editors from being accused of taking sides in personal and professional quarrels. It is also the reason that reviewers themselves prefer to remain anonymous. No scientist wants to suffer the professional and personal consequences from either refusing or accepting a paper they should not have refused or accepted. It is also why peer review is a superficial review. The reviewers do not wish to be dragged into the minutia of scientific debates and quarrels. Instead, they concentrate on the basics that everyone can agree on.
In the end, it is the anonymous and secret nature of the peer review process that marks it as not part of actual science. The entire point of science is that all observers of a phenomenon can agree they see the same thing. Critical to creating that agreement is ruthless transparency. Secrecy is antithetical to the functioning of science, and peer review is a secret process. Science is not settled by the secret complaints of the anonymous.
Mere peer review should never be the basis of public policy, because when you get right down to it, we have only the word of the journal’s editor that a peer review was even performed. There is no formal mechanism to assure that peer reviewed is performed or that the reviewers have the competence to review the paper in question. If the journal’s editor is corrupt, then there is no independent mechanism that forces a peer review or ensures its quality. The entire system is based on a presumption of trust and on the discipline of the free market in scientific publishing. If an editor wanted to fake a peer review, no one could tell for a very long time.
Even if everyone is honest, the inevitable professional biases of peer reviewers can cause them to reject papers that call into question the tenets upon which the reviewer’s own work rests. If a scientific field is relatively small and all the peers share the same scientific blind spots or misapprehensions, then peer review can’t catch even gross errors that become obvious in hindsight. It is common for peer reviewers to repeatedly reject papers that substantially alter a major tenet of a field. Most of the game-changing papers of the last century were rejected by multiple peer reviewers at multiple journals.
People who try to defend a scientific assertion by claiming it appears in a peer reviewed journal are making the weakest defense possible for the assertion. All it means is that some editor and the reviewers he selected thought it met their minimum quality standards for publishing. Once you raise the specter of political corruption on the part of editors and peer reviewers, it doesn’t even mean that.
Replication and proven predictive power, not the opinions of peers, test science assertions. Those iron objective tests separate science from all other disciplines. In the long run, scientific peers are always wrong. Science does not advance by consensus, it advances by the repeated destruction of each generation’s existing consensus.
If the strongest defense someone can muster for a paper’s conclusions is that “it has been peer reviewed,” that is a dead giveaway that the paper itself is very, very weak. Basing policy on scientific studies that have been merely peer reviewed is just flat insane.
I’ve reviewed only a few manuscripts for a relatively small journal (in my specialty area) I don’t have an extensive CV and don’t publish much – my main area of work is clinical service and teaching. I can speak to my experience, only, but you know, peer-review is just a bunch of people reviewing submitted manuscripts. Peer review CAN be rigorous (I hope I’ve done due diligence in my case, I rejected most manuscripts or suggested major changes), but that depends on the reviewers and the editors. Anyone else want to chime in? Like all human activity, peer-review may be rigorous, sloppy, or any point in-between.
Shannon – I’m not sure anonymity of the reviewer is such a bad thing, although I tend to think transparency is always better. If you are a junior member of a faculty, and reject the paper of someone higher up in the food chain, even not at your institution, could it hurt your career? How about you play a political game and approve papers by the bigger fish so you can be treated more fairly? I don’t know. Again, what do others think?
Onparkstreet,
I’m not sure anonymity of the reviewer is such a bad thing,…
It’s not a bad thing… for publishing. It’s a very good way to keep scientific journals alive and credible. It just doesn’t have anything to do with science.
Oops, my above comment was confusing: you are not going to be reviewing others at your own institution.
Oh, I see your point, now.
So, have disclosures at the end of each paper – who reviewed the paper and then perhaps raw data could be provided, if asked, by appropriate sources? Are there journals that do this already? It could place the journal at a competitive advantage.
Truth in science is established by replication. If you don’t like what Galileo is saying about the motion of the heavenly bodies, get your own telescope and look for yourself. If you think Aristotle was write and Newton was wrong about the laws of motion, make an inclined plane and get a stopwatch and check it out.
Ideally, the purpose of peer review should be (a)to ensure that journal space (and, more importantly, the attention of journal readers) is not wasted on stuff that is clearly worthless, and (b)to provide sufficient data for another experimenter to attempt to replicate the results.
Peer review is no more a certification of truth than the willingness of a court to hear a lawsuit is a certification that the suit will prevail…It is no more a certification of truth than the willingness of a corporate executive board to listen to a project proposal is a certification that they are going to approve it.
“If you think Aristotle was write”
Blog commenting kills brain cells. I need neither peer review nor replication to establish the veracity of this great truth.
In 1899, the Mayo brothers submitted a paper to the American Journal of the Medical Sciences, the premiere American journal of the time, with 105 cases of gallbladder surgery. The editor, Alfred Stengle, asked around Philadelphia to see how many such cases had been done in the city. He then returned the paper to the Mayos unpublished. The early history of not invented here syndrome. By 1903, major European surgeons were traveling to Rochester to see them work but Philadelphia was not interested. There is nothing new under the sun.
I once had a small paper rejected by a journal because I had not cited the reviewer’s paper on the same topic. He later told me he had rejected it. He wasn’t the least sheepish about it. He told me it served me right.
http://en.wiktionary.org/wiki/peer
Tell me, from which definition does the term “peer review” draw?
And how did the English language manage to get the same word to mean both “equal” and “more equal than others?”
Another thought: in another blog’s comments I invoked the Killian document controversy – it wasn’t peer review that revealed the forgery.
(Citing this example is not without irony; Rathergate’s chief non-peer reviewer is one of the blogosphere’s top CRU defenders.)
I believe you meant. to say minutiae instead of minutia.:-)
Great post and VERY timely, as the AGW set try to convince everyone that ‘peer review’ = ‘cosmic truth’ rather than “peer review” = publishable
“Tell me, from which definition does the term “peer review” draw?”
“And how did the English language manage to get the same word to mean both “equal” and “more equal than others?””
It means the first, but both have same root idea. A jury of ones peers means a jury of ones legal equals. The House of lords are made up of peers, legal equals not to commoners, but to each other. This is why until the 20th century, members of the house of lords could only be convicted by the house acting as their jury.
Peer review may not be perfect, but as the author states, in most cases the reviewers will certainly check for good methodology, valid assumptions and valid logic. They will also in general be picky about error analysis. They will try to make sure that the conclusions do not contradict the assumptions. In other words, peer review is pretty efficient at getting rid of the pure junk. Yes, there will unfortunately also be a resistance by some to novel ideas – but that is true with or without peer review.
You make the argument that holding up a single paper as truth based on peer review is foolish. That’s true, but that is a straw-man argument and that’s not what we’re talking about. When you look at a large body of work across disciplines that all support the same conclusion then you can make a very strong statistical argument in favor of that conclusion. You have to look at the big picture. And the big picture says that there is a mountain of evidence in support of AGW, and very little credible evidence against it. You can’t extrapolate your argument to the case we have here. That’s like basing your conclusion on just a few data points. I’m afraid your article would not be likely to pass the peer review process without modifications.
Jeb,
You make the argument that holding up a single paper as truth based on peer review is foolish. That’s true, but that is a straw-man argument and that’s not what we’re talking about.
Unfortunately, this is not a strawman. I wish it were.
The main argument put forth by for the validity of much of the work on CAGW, especially by lay advocates, is that it has been peer reviewed. Conversely, they dismiss the work of skeptics by pointing out that they may not be peer reviewed by climatologist. Pick a climate debate thread or even a news story at random and see how often the phrase “peer reviewed” crops up. I think it very clear that in the public mind of warming alarmist, peer review as come to mean “independently confirmed” of worse. “has proven predictive power”.
This is simply a type of argument from authority. It says, “this work must be valid because the other scientist/philosophers/priest/shaman say it is.” However, the scientific method exist precisely because arguments from authority are useless.
You have to look at the big picture. And the big picture says that there is a mountain of evidence in support of AGW, and very little credible evidence against it.
Except the mountain is just the same molehill pictured over and over from many different angles. The overwhelming majority of evidence for CAGW comes from just four major groups and the work of under 50 individual scientist. That’s a very small number of people and makes dangerous groupthink very likely. Moreover, this same small tight knit group is likely involved in peer reviewing each others papers. So, basically, you have a small number of people, most of whom have staked their reputation on CAGW being accepted, all reviewing each other’s papers and finding out that gosh all those other scientist agree with them.
This type of problem has arisen time and time again in science. Entire fields have wandered down the wrong path for decades because everyone in the field held the same unexamined assumptions.
And of course we have the killer fact that none of this research has proven predictive power which is the acid test for science. All this is very small, dubious and untested data on which to make decisions that will kill hundreds of millions of people.
I’m afraid your article would not be likely to pass the peer review process without modifications.
I would hope not as this is not a scientific paper and makes no claims to being so. If it passed peer review in a scientific journal then things would be very wrong indeed.
While I enjoyed your article, and agree with several parts of your analysis of journal peer review, I think you mischaracterise the process in a couple of places.
Why then do we use peer review? Simple, publishing glaringly flawed papers or being seen as taking sides in a scientific dispute destroys journals both professionally and economically. Peer review protects the journals and the careers of the people who staff them.
A more benign interpretation of this is that peer review can weed out manuscripts that are demonstrably flawed, or incorrect, and this helps prevent more ‘poisoning the well’ of collective understanding than would otherwise occur. By assuming that the ‘worst’ papers are discarded by this process (particularly so by the better journals) we can arguably make faster progress by considering that what is published is at least somewhat likely to be correct. There is always the opportunity to demonstrate that a published paper’s conclusions are incorrect at a future date, after all.
If there were no such filtering, then how would we establish a system of trust in the reported work of others? This would likely arise naturally, out of efficiency, to identify significant work so that we could extend it or carry its conclusions forward? We would still then rely on the arguably even more politically suspect ‘word of mouth’ of our peers, or we’d have to resort to repeating interesting work directly, potentially wasting time and resources. Journal peer review not just a matter of protecting the careers of individuals, or the status of publishers, and it isn’t perfect, but it can be an improvement over no peer review.
Peer review protects a journal’s reputation by hiring experts in a field to check papers prior to publication.
I understand the definition of ‘hiring’ to involve contracting or payment. I can’t think of a journal in my field that pays reviewers (I’ve certainly never been paid), and I can think of only one that offers more than an honorarium to editors, and in my experience most do not even offer that.
This blame passing also keeps journals and editors from being accused of taking sides in personal and professional quarrels.
It doesn’t. I have known of (usually unfounded) accusations of bias and side-taking in reviewer selection.
It is also the reason that reviewers themselves prefer to remain anonymous. No scientist wants to suffer the professional and personal consequences from either refusing or accepting a paper they should not have refused or accepted. It is also why peer review is a superficial review. The reviewers do not wish to be dragged into the minutia of scientific debates and quarrels. Instead, they concentrate on the basics that everyone can agree on.
I prefer to remain anonymous as a reviewer because scientists are human, and humans – even if they think they are above them – usually have emotional responses even to intellectual endeavours. I think that most manuscript authors truly believe that what they write is correct. The natural human response to criticism (and you often see this in authors’ responses) is to be defensive, and this can lead to the creation of petty tit-for-tat personal interactions even if the reviewers’ criticisms are true and valid. That’s not restricted to science, either.
Contrary to your article, I would say that anonymity actually encourages me to be drawn into the detail of the scientific arguments of those manuscripts, because anonymity subverts arguments from authority: the author cannot pretend or assert that they are an overriding authority, and must consider the reviewer’s arguments themselves, rather than merely disparaging the reviewer. Furthermore, many reviewers, including myself, would rather see a borderline paper published – for discussion, and because reviewers generally accept that they may well be wrong – than rejected.
As it happens, much direct criticism of work occurs at conferences, and I have witnessed and taken part in very vigorous disagreements between individuals who were clearly not anonymous to each other. Many junior scientists – who could have valid and decisive input – choose not to become involved publicly, for the same reasons as above (and not just cynically, because they want a job; presumably they would prefer to work for the scientist they consider correct, and support their position in a public argument, in any case.)
In any case, journal peer review is only that: peer-review for publication. You are correct to point out that neither peer-review nor publication necessarily establish or confer ‘truth’. You are also correct to point out that it is the repeated testing of claims against nature that establishes this ‘truth’. The repeated testing you describe is, I believe, the true process of scientific peer review as distinct from journal peer review: that those who are capable of testing the assertions may do so.
Finally, just as it is not sufficient to point out that work is ‘peer-reviewed’ and assume that this brooks criticism, it is likewise not enough to point out that journal peer review is broken, or otherwise suboptimal, for a piece of work, and assume that this allegation throws the results into doubt. If that argument is made, we have to accept that it is broken or suboptimal for all science, and not just the science we disagree with.