Megan McArdle posts about the shallow and imprecise study done to justify the insane symbolic tradeoff between zero preservation of the ozone layer versus dead asthma sufferers. One commenter so precisely explains the real world challenges of creating a statistically valid study that I feel justified in reproducing it here in its entirety.
■Small sample: the smaller the sample, the harder it is to find an adverse effect. That’s why drugs like Vioxx made it to market: distinguishing problems from background statistical noise needed a lot of patients. I know more than one analyst who argues that medical studies are generally too small–because humans are so variable, they don’t reliably pick up any but the strongest effects.
Just curious – Since finding and examining enough patients is the biggest expense behind developing drugs, are you willing to increase the cost of future drugs by 20-30% to have a larger sample size? 40-50%?
I’m intimately involved in this process, and it’s unbelievable to watch these study protocols get designed….as an example:
Study X is for Chronic Obstructive Pulmonary Disease. It needs 2000 patients with moderate-to-severe COPD who will fail a POST-bronchodilator breathing test (i.e. even after they get a drug, they still can’t breathe at 70% of normal).
The patients can’t have any confounding major medical conditions.
Patients in this population tend to have received alot of drugs which start to cause eye problems. But patients in this study can’t have any existing eye problems, so that they can study whether the investigational drug causes such problems.
The patients have to agree to several 12-hour long study visits, agree to coming off all of their current medications for at least 4 weeks prior to receiving study drug. Or placebo. Let’s not forget that they might be required to take a placebo.
They might have a exacerbation while they stop taking their medication, and the exacerbation might be enough to hospitalize them. But since they haven’t started taking the study drug yet, their hospitalization wouldn’t be covered by the study sponsor.
They won’t really get any compensation for their participation, because it can’t be sufficient to make them WANT to participate. That’d be unethical.
And they won’t be able to get the study drug after their participation ends, because it’s investigational.
So basically, you need to find people with a significant illness who are willing to put themselves through hardship, sacrifice, and possible severe health problems for the sake of altruism.
Do you have any idea how hard it is to find 2000 people fitting that profile? The time and money involved in simply IDENTIFYING those people?
To give you an example, I just spent about $1 million finding 50 such individuals. That doesn’t include conducting the actual research. That’s just creating awareness, pre-screening them, and connecting them with the doctor’s office.
And this is just one study….you need several such studies to get a drug approved. And many more studies before that to show the drug is worth taking into a 2000 person study.
My point is: People have no idea what it takes to conduct such research, and why drugs are so commensurately expensive. If you want less statistical noise, be willing to pay for drugs that are much more expensive.
The ugly secret of modern medicine is that for many drugs and procedures we don’t have statistically significant population sizes to conduct truly valid studies of them. Neither do we have the ability to test for all of the possible drug combinations that real-world patients have to take. We’re guessing a lot more than we would like to admit in many cases.
We get bombarded by conflicting medical information, in large part due to the practical limitations of medical studies. The statistical significance of many studies is dubious due to their population size even before you get into questions of individual variations of the real people in the study. For example, few studies actually track the race of individuals involved and just assume that all races respond to medications the same, even though we know that is often not true. Yet chopping up the already limited study population into even smaller groups based on race completely destroys any statistical significance the study might start with.
We need to reexamine how we conduct studies and how we process their information in medical and political decision making. For starters, we could start paying subjects to compensate them for the risk they take for the common good. After all, we pay soldiers. Why are people who put their lives on the line battling disease any less deserving of compensation than those who battle human evil? I doubt that the desire for financial gain will distort studies more than does the dearth of subjects we currently have. We also need an easy to understand system of grading studies for the media based on their statistical and methodological soundness. For example, we could simply grade studies as being “A”, “B” or “C” grade starting with large, long-term studies, and going down to studies with a dozen subjects over the course of week. That way, people could see at a glance how sound a study’s results are likely to be.
I’ll say it again. Having bad data is worse than no data at all. This is especially true when political symbolism drives the creation of the data. We can expect to see more and more of this as the Left tries to use a veneer of shallow science to justify policies they’ve already decided on.