Much of what medical researchers conclude in their studies is misleading, exaggerated, or flat-out wrong. So why are doctors—to a striking extent—still drawing upon misinformation in their everyday practice?
The arguments presented in this article seem like a good if somewhat long presentation of the general problem, and could be applied in many fields besides medicine. (Note that the comments on the article rapidly become an argument about global warming.) The same problems are also seen in the work of bloggers, journalists and “experts” who specialize in popular health, finance, relationship and other topics and have created entire advice industries out of appeals to the authority of often poorly designed studies. The world would be a better place if students of medicine, law and journalism were forced to study basic statistics and experimental design. Anecdote is not necessarily invalid; study results are not necessarily correct and are often wrong or misleading.
None of this is news, and good researchers understand the problems. However, not all researchers are competent, a few are dishonest and the research funding system and academic careerism unintentionally create incentives that make the problem worse.
(Thanks to Madhu Dahiya for her thoughtful comments.)
models do not create data … they need to stop acting like they do …
That is an excellent article and I know it is true from my own experience. One example:
When I was a surgery resident 40 years ago, the standard handling of a colon cancer was what was called “The no touch technique.” It was a result of a study published By George Crile Jr, the son of the founder of the Cleveland Clinic. “Barney” Crile was also well known for several other iconoclastic studies, including one that recommended simple mastectomy for breast cancer. Crile was on all the women’s TV shows of the time.
I was at a College of Surgeons meeting on cancer about 1973, when a group presented a startling paper. It was using the absolute opposite of Crile’s technique. He recommended not touching the tumor until the blood vessels had been cut so no cancer cells would be dislodged and float up into the liver. It made sense so was widely adopted. The group presenting the new study, had injected chemotherapy into the colon near the cancer. In order to allow the drug to reach the veins and the liver, they did NOT cut the blood vessels until the colon was completely freed from its attachments.
What was truly startling was that they had a control group in which they had NOT injected chemotherapy. That was almost an ethical lapse as everybody “knew” that handling the tumor would spread the cancer.
What they found was that the control group in their study had the same results as the “no touch” group in Crile’s paper. How could this be ? They asked for Crile’s rough data from Cleveland Clinic. They found that his control group results had NOT been adjusted using time/life tables. The treated group had been.
Of any group of cancer patients, a certain percentage will die of other causes during the usual five year followup. For that reason everyone (but Crile) used numbers corrected for the expected survival. Raw survival will be worse. He hadn’t done it and his results were a fraud.
What was also startling was the presence of a famous Cleveland Clinic colon cancer surgeon (who had NOT been part of the study) in the audience. He was asked to comment before the audience of about a thousand surgeons. He declined, saying “You will have to ask Dr Crile.”
In later years, I knew a resident from Cleveland Clinic who told me privately that women with tiny breast cancers were referred to Crile while more advanced cases with larger tumors were referred to the other surgeons at the clinic. His results, which were EQUAL not superior to the other surgeons, were on selected cases with better prognoses.
I have many other stories if anyone is interested. Part of the problem is that, as you see, it is more complicated to explain why the results are wrong. One thing I learned at Dartmouth is that, if a certain treatment is universally used, it is probably correct. The variation in treatment means uncertainty about what is best. Of course, much research is conducted on those areas where the treatment is uncertain.
Oh, and just to show that error never dies, here is an article from next month using “no touch technique”
There were a lot of meta-studies promoting cholesterol reducing statins for Alzheimer’s.
Now some are coming out and saying the opposite is true – statins hasten Alzheimer’s.
It turns out that cholesterol is needed for proper brain functions. The whole low fat diet movement is shrinking our brains, and statins are additionally draining compounds that are necessary for proper health.
The real dietary problems are refined sugars.
Horror story.
http://www.independent.co.uk/life-style/health-and-families/health-news/exposed-edward-erin-the-doctor-whose-faked-asthma-drug-test-results-proved-fatal-8662548.html
One way to deal with it?
http://www.independent.co.uk/news/science/the-bad-science-scandal-how-factfabrication-is-damaging-uks-global-name-for-research-8660929.html
Wakefield has hundreds of deaths on his conscience, assuming he has one, for the autism scare and the kids dying of preventable diseases like measles.
The Lancet has also suffered a great deal (rightfully) for its fake Iraq death study and Wakefield.
A very interesting read that applies to almost every research field (not just medicine). Of course, recognizing that there is a problem, or many of them, doesn’t tell us that they can be solved. Unfortunately, I don’t think anyone can come up with a “better” system than the current one. The problems, of course are several:
1) Statistics can quickly become far too complicated for most laymen and most researchers who don’t specialize in statistics. Teaching “more stats” is probably not going to solve the problem, because you’d have to teach “a lot more” than most people can handle (certainly most MDs, journalists or lawyers, who may not have much math background)
2) The problem every study faces is that as sample size gets bigger, the likelihood of finding significant correlations increases, but power decreases. Of course, it is up to the researcher to do a substantive interpretation for the reader. But what does a substantive interpretation mean in medicine, when you may be dealing with lives?
3) Statistical artifacts will always exist in every study. The article mentions that many of the 49 famous studies were shown to be “false or significantly exaggerated”. Well, significantly exaggerated itself is a statistical artifact. It doesn’t tell a practicing MD much. Also, a meta-analysis is itself a study of statistical artifacts. I.e., you can’t get away from it, and there is so much wiggle-room in most statistical interpretations, that you will always have this problem.
4) The emphasis on “newness” in publishing isn’t necessarily true, or a bad thing. There are journals that specialize in publishing new ideas, and journals that specialize in publishing in methods. Of course, the problem is the audience; MDs, journalists and lawyers aren’t going to read the methods journals. That is never going to change.
So how do you fix it? Can you fix it? I’m not sure anything more can be done without creating additional unforeseen problems down the road. Most of the problem, seems to be with the audience, not the scientific community. Most researchers are well aware of these problems, which is why whenever we read any study, we are always skeptical and never take the statistical outcome to be authoritative, just interesting.
PS: Also, there’s the additional problem of what gets published, and how it gets through the review process. Everyone knows there’s politics involved, but there are politics involved in every human system. No way around that. We all aspire to objectivity, but no human system can ever be totally objective. So that criticism, isn’t really a criticism.
Also, every study has some flaws. Every study has limitation on what variables are controlled for, how the study is conducted etc. You try to carry out all sorts of validity checks, but all validity checks in statistics are to a certain degree, subjective. So you can’t really fix that problem either.
The only thing that can be done, is to retest the results more often. The problem is, if it takes 5 years to do a study of that sort, it takes 5 years to retest. That’s the biggest limiting factor. It does take a long time to overturn wrong results, because it takes a long time to do these studies.
Can anyone really fix this?
An academic statistician friend told me that there were three problems with medical research projects and stats.
(i) The medics typically wanted a statistician to analyse the numbers from a study they’d designed themselves, rather than have him be part of the team who designed the study.
(ii) Having called in the statistician too late, they would typically be reluctant to reward him by making him a co-author of the resulting papers.
(iii) But nor would they be prepared to reward him with a consulting fee either.
The problem is largely not one of the consuming audience, the practicing doctors. It is the providers of such studies. The pressure to publish and the power and influence of the journals. In some specialties there is also the influence of device makers on funding.
It’s a long story but I went to Dartmouth after I retired to learn methods. One topic I chose is the shunt that is used to connect dialysis patients to the machine three times a week. At the time (1994), the most common method was to use a prosthetic graft. I believed that using the patient’s own vessels, the arm veins, was better but a bit more difficult. Who would fund a study to show the artificial graft was less effective ? The graft makers ?
As in most th8inkgs….follow the money.
” I believed that using the patient’s own vessels, the arm veins, was better but a bit more difficult. Who would fund a study to show the artificial graft was less effective ? The graft makers ?”
How about those who pay? The insurance companies, or the hospitals. Plenty of sources, if the intent is to show how something can be done cheaper. I’m not so sure this is about “the money”. Plenty of government grants floating around to do just that.
Retraction Watch is worth a weekly/monthly visit…
http://retractionwatch.wordpress.com/
“How about those who pay? The insurance companies, or the hospitals. Plenty of sources, if the intent is to show how something can be done cheaper. I’m not so sure this is about “the money”. Plenty of government grants floating around to do just that.”
All ESRD patients are Medicare after 6 months. Insurance companies can’t be blamed for this one. The only source of funding would be the NIH division of kidney diseases. That’s a long line . We were studying a new methodology that was almost completely done using Medicare claims data, of which there is (or was) a 100% sample unlike most Medicare claims. An ESRD patient has to have a transplant or dialysis or die. For that reason, there is 100% followup. We put together a team for the grant application including an economist from Dartmouth who worked on health economics and a statistician. I was the clinician.
I submitted the grant application and waited. Finally, a sympathetic editor who liked the idea, leaked to me the reviews of the peer reviewers. They didn’t understand the method and were more used to small clinical series, as most of the medical literature is. We were proposing to study all the claims submitted to Medicare for 7 million patients for five years. The database would be huge but we had minicomputers (this was 1995) and SAS software to run the study.
We didn’t get the money. Outcome studies like this are more common now but I got discouraged after this experience. I wasn’t supporting my self with this. It was a post-retirement project and I hoped to eventually have a second career doing studies attempting to measure quality.
One problem might have been that a group at U of Michigan were a big contractor with Medicare for all research with ESRD. They weren’t interested in the surgery aspect I was but they got most of the money. We (the economist and me) estimated the possible savings to Medicare of about a billion dollars. It would have meant more training for dialysis center folks and a differential fee schedule for the surgeons doing the shunts as the vein shunts are more difficult. We had included those items in the estimate. It’s still not be done and it’s been 20 years. However, others have come to a similar conclusion and vein shunts, called “fistulas” are more common now.