Does this sound familiar ?

by

The science community is now closing in on an example of scientific fraud at Duke University. The story sounds awfully familiar.

ANIL POTTI, Joseph Nevins and their colleagues at Duke University in Durham, North Carolina, garnered widespread attention in 2006. They reported in the New England Journal of Medicine that they could predict the course of a patient’s lung cancer using devices called expression arrays, which log the activity patterns of thousands of genes in a sample of tissue as a colourful picture. A few months later, they wrote in Nature Medicine that they had developed a similar technique which used gene expression in laboratory cultures of cancer cells, known as cell lines, to predict which chemotherapy would be most effective for an individual patient suffering from lung, breast or ovarian cancer.
 
At the time, this work looked like a tremendous advance for personalised medicine—the idea that understanding the molecular specifics of an individual’s illness will lead to a tailored treatment.

This would be an incredible step forward in chemotherapy. Sensitivity to anti-tumor drugs is the holy grail of chemotherapy.

Unbeknown to most people in the field, however, within a few weeks of the publication of the Nature Medicine paper a group of biostatisticians at the MD Anderson Cancer Centre in Houston, led by Keith Baggerly and Kevin Coombes, had begun to find serious flaws in the work.
 
Dr Baggerly and Dr Coombes had been trying to reproduce Dr Potti’s results at the request of clinical researchers at the Anderson centre who wished to use the new technique. When they first encountered problems, they followed normal procedures by asking Dr Potti, who had been in charge of the day-to-day research, and Dr Nevins, who was Dr Potti’s supervisor, for the raw data on which the published analysis was based—and also for further details about the team’s methods, so that they could try to replicate the original findings.

The raw data is always the place that any analysis of another’s work must begin.

Dr Potti and Dr Nevins answered the queries and publicly corrected several errors, but Dr Baggerly and Dr Coombes still found the methods’ predictions were little better than chance. Furthermore, the list of problems they uncovered continued to grow. For example, they saw that in one of their papers Dr Potti and his colleagues had mislabelled the cell lines they used to derive their chemotherapy prediction model, describing those that were sensitive as resistant, and vice versa. This meant that even if the predictive method the team at Duke were describing did work, which Dr Baggerly and Dr Coombes now seriously doubted, patients whose doctors relied on this paper would end up being given a drug they were less likely to benefit from instead of more likely.

In other words, the raw data was a mess. The results had to be random.

Duke had … chosen, within a few months of the papers’ publication (and at the time questions were being raised about the data’s quality) to launch three clinical trials based on their work. Dr Potti and his colleagues also planned to use their gene-expression data to guide therapeutic choices in a lung-cancer trial paid for by America’s National Cancer Institute (NCI). That led Lisa McShane, a biostatistician at the NCI who was already concerned about Dr Potti’s results, to try to replicate the work. She had no better luck than Dr Baggerly and Dr Coombes. The more questions she asked, the less concrete the Duke methods appeared.

Now we have two “skeptics” wondering about the data. Here it begins to acquire a marked similarity to another scientific controversy, although one with much larger world wide consequences.

In October 2009, officials from the university arranged for an external review of the work of Dr Potti and Dr Nevins, and temporarily halted the three trials. The review committee, however, had access only to material supplied by the researchers themselves, and was not presented with either the NCI’s exact concerns or the problems discovered by the team at the Anderson centre. The committee found no problems, and the three trials began enrolling patients again in February 2010.

Some of us call that a whitewash and it also bears some similarity to another such committee investigation in 2010.

Subsequently, the committee’s members interviewed Dr Baggerly about the problems he had encountered trying to sort the data. He noted that in addition to a lack of unfettered access to the computer code and consistent raw data on which the work was based, journals that had readily published Dr Potti’s papers were reluctant to publish his letters critical of the work. Nature Medicine published one letter, with a rebuttal from the team at Duke, but rejected further comments when problems continued.

Other journals that had carried subsequent high-profile papers from Dr Potti behaved in similar ways. (Dr Baggerly and Dr Coombes did not approach the New England Journal because, they say, they “never could sort that work enough to make critical comments to the journal”.) Eventually, the two researchers resorted to publishing their criticisms in a statistical journal, which would be unlikely to reach the same audience as a medical journal.

Now it is getting really familiar to some of us. Journals were blocking access to critics of a paper that was considered a major advance. Grants were involved and, potentially, lucrative commercial ties.

The university’s lapses and errors included being slow to deal with potential financial conflicts of interest declared by Dr Potti, Dr Nevins and other investigators, including involvement in Expression Analysis Inc and CancerGuide DX, two firms to which the university also had ties. Moreover, Dr Califf and other senior administrators acknowledged that once questions arose about the work, they gave too much weight to Dr Nevins and his judgment. That led them, for example, to withhold Dr Baggerly’s criticisms from the external-review committee in 2009. They also noted that the internal committees responsible for protecting patients and overseeing clinical trials lacked the expertise to review the complex, statistics-heavy methods and data produced by experiments involving gene expression.

Now, the commercial connections are all, according to the alarmists, on the global warming skeptic side but some of us doubt this. The more we have been involved in research, the more we doubt that version of the story. Trillions of dollars, and Euros, and Yen are at stake. The grants and the other goodies available to researchers who come to the correct conclusions are in a similar scale.

The difference here is that the dubious results of flawed or faked research can be tested against real world consequences. So far, the global warming models have not been tested and the honesty of the research workers is all we have. And we know what that is worth.

17 thoughts on “Does this sound familiar ?”

  2. If taxpayer funded grants are involved, failure to release data should should lead to investigation, fraud charges and jail if guilty.

    Science has a problem with a politicized establishment, which seems indifferent to the rigorous requirements of the scientific method. On Global Warming there is a laughable focus on computer models which have been unable to predict anything. This is Cargo Cult Science. In twenty years this will looked at as more ridiculous than Lysenkoism.

    See Feynman:
    http://www.lhup.edu/~DSIMANEK/cargocul.htm

  4. A lot of academic research is corrupt. Remember the mess at the East Anglia University on man made global warming? They wanted to change the facts to suit the hypothesis.

    Duke hasn’t really been known for debated and logically thought out positions lately – just ask the lacrosse team ;-)

  6. The point of the post is to suggest that the global warming situation is not as rare as implied. The AIDS story is another tale of fiasco. I pointed out a few years ago that the truth about who actually discovered the AIDS virus was complicated by scientific fraud. That fraud cost lives instead of money, as is the case with AGW. Fortunately, in that case, the truth finally came out but it took a decade.

  8. Wonder how much the problem is exacerbated by gov’t money.

    Some of it is simple ego. Gallo was already on the US payroll but there was a patent involved. The Human Genome Project was government funded and was expected to take ten years. Craig Venter used private funds and did it in two years. He is very unpopular with people like Watson, of Watson and Crick. Crick did all the work with DNA and Watson thought he had a ten year sinecure with the Genome Project until Venter showed up.

    This is just not that rare. I had an editor refuse to publish a paper of mine because I had not listed a paper of his in the footnotes of a previous paper.

  9. You call them skeptics, we call them scientists.

    Cancer, and cancer drugs, are huge potential moneymakers.

    Curiously, the greatest moneymaker in the whole climate science debate would be the opposite–a scientist who offered a convincing debunking of any of the systems that undergird the scientific consensus would be a hero–could write his own ticket, could cash in, could trigger a cascade of related projects. Plus, there are dozens of concurrent systems of study that relate to climate change and the manmade conclusion. Maybe hundreds. Tundra, sea ice, straight temperature measurement, on and on–all concurrent, parallel. If there were problems in the temperature conclusions, the causality systems, or the extension that dares to lay the blame on man, it should be easy, easy to make the case. An easy prize to seize.

    Why hasn’t it happened?

    The handful of refutative or noncorroborative studies have fallen apart, melted away. It’s like a pinata the size of an aircraft carrier that nobody seems to be able to hit, while a few thousand whacked-out loudmouths declare that AGW is dead. Meanwhile, all the cheating, lying, money-sucking AGW-whores have somehow been cleared of any wrongdoing, sometimes two or three times; the studies and observations keep piling up, the corroborations keep mounting. The pinata gets bigger and bigger, full of goodies–if it’s so delicate, well, gentlemen, whack away. But you can’t. You have no science, nothing but pathetic arguments like the one above–some Duke researchers screwed the pooch, then tried to avoid getting what they deserved. That proves, what exactly? Give me a minute, you say. Here, I’m going to knock that pinata wide open! Oh, hey, my stick doesn’t work, it’s (mumble), but if I hit that pinata it sure enough would split wide open, here we go, I’m going to hit it…hey look! IPCC e-mails!

    ice

  10. the corroborations keep mounting

    Please be specific. All you have given us is a list of assertions and some personal shots at “whacked-out loudmouths” you disagree with. I wouldn’t trust someone who argues the way you do to fix my car. Why should anyone trust your arguments about weighty matters of public finance?

    AGW theorists propose colossal increases in public spending and restrictions on freedom to deal with problems whose future effects continue to be highly uncertain. The burden is on them to show 1) that the cost/benefit tradeoff is favorable and 2) that the magnitude of the likely benefit is so big that it justifies massive public spending that will crowd out known lifesaving expenditures on (for example) medical R&D, malaria eradication, education and national defense. So far the AGW theorists haven’t come anywhere close to doing this. They are like doctors whose initial response to a patient with vague symptoms is to recommend major surgery. “First do no harm” applies in public affairs too.

  11. The essence of science is to predict phenomena. This “climate science” is unable to do, and the public, being a little slow, but not stupid, is catching on to this. Interesting that Global Warming has been ditched for Climate Change, which is harder to falsify. Buddy, this has scam written all over it.

  13. “… the global warming models have not been tested …”

    Then they are junk. Period. Full stop.

    One way to tell whether someone really understands modeling and simulation (two very different things) is their attitude toward Independent Verification and Validation (IV&V). Professionals won’t accept anything from M&S that hasn’t been through IV&V. Hacks, frauds and incompetents don’t see any need or actively resist IV&V.

  14. Jonathan: I have no comments regarding public sector or government initiatives to combat global warming. I made no reference to public finance. I understand AGW, and study it closely. There’s no question of whether I believe it or am persuaded; it’s a scientific question so belief mechanisms are not relevant for grownups. The weight of evidence is overwhelming so far, and I await the new studies, or those that will be published by somebody who doesn’t choose to resign in disgrace once the pathetic wingnuttian methodology of the study is fully exposed.

    Since you opened the topic of public policy, I’ll say that I agree with you. Many of the measures proposed so far are scary. I’m deeply worried about the potential economic costs. In fact, it’s correct to say that I hope the science is refuted. But–try to follow this–the scary cost of some proposals, whether accurate or overblown or underplayed, does not have any effect on the validity of the science on AGW.

    To put it in another way you brought up–I wouldn’t offer to work on your car, for the simple reason that your car is not broken. Some say it burst into flames and immolated your Peter Frampton 8-tracks, but Peter Frampton smoked dope, which means there’s no such thing as fire, so the car is not a burned-out shell chairoscurro’d with melted Cheeto bags. It runs just fine, let’s go for a drive and look at our prosperous future.

    We can’t discuss the solution, you say, because it’s too much money and it cuts into the freedoms I am suddenly keen on preserving.

    I’ll take you at your word that you favor spending on medicine and so forth, though the malaria point quivers like an arrow in a tree, just begging us to detour to the next bogus wingnut fake-science trope, the weeping over the mistreatment of the much maligned DDT. Anyhow, I’d love very much to lose the scientific debate over AGW. It’s going to cost a bundle and probably accelerate the ascension of China and India, and I hate it. But hating it doesn’t make the problem go away. To make the problem go away, your ilk will need to participate successfully in a scientific argument over AGW that doesn’t involve preaching to the very angry, deluded choir and waving away the rules. At present no such argument exists. Your post above is nothing like a scientific argument; it’s a sidetrack. On the other hand, if we accept the premise that AGW is BS, then all the fearful flouncing over massive public expenditures isn’t relevant either.

    But if it isn’t BS, then the fearful flouncing is relevant. An intelligent political conversation would consider those massive public expenditures, and we’d get somewhere. But we ain’t. We’re calling people socialists and stuttering on about freedoms, while in pretty much every other political area you–the same wingnut folks right on up to igloo-building senators–are perfectly happy to slurp socialist teat and trammel the tar out of various freedoms.

    So I will concede the massive public expenditures, just as soon as you concede that, science wise, you got dick. Specificity isn’t required, if you can be persuaded or compelled to drop the distracto-jump into public policy. There simply is no evidence that counters key observations on climate change (including the link to the tired old cycle “it’s normal” argument someone gave me above, participating in that most cherished of wingnut scientific tactics of citing ones own previous work because links are proofy-proof. Grownup scientists have more than USA Today charts, and Woods Hole has disavowed those conclusions, and blah blah–you don’t do science, so why am I bothering?) All collapsed, all refuted, withdrawn, trashed. Spencer and Braswell, Plimer, Wegman, that gassy ignoramus Sir Monckton of Not-a-Lord on Shitwick, Lindzen and Choi (twice), etc. etc. etc. Such a huge, slow-moving zeppelin of a target, AGW should be easy to discredit and bring down in a flaming mess–but no luck so far. Why is that? Don’t know. All you can do is

    1. Swear to God Obama is going to put you in jail for having an incandescent light bulb. Don’t shoot him, but if he gets shot well he asked for it, coming for my guns and light bulbs and all.
    (And, as always, the failure to call the person who makes the argument above a lunatic. He’s easy to find–he’s 92 members of the US House of Representatives. I await your response.)

    2. Declare that all science, except that which you trust, is untrustworthy, because of money, natch–I read it from some scientists who were able to get the word out, thank god that the oil and coal lobby still believe in neutral, objective journalism. (My well-poisoning tu quoque is ironic. Concede that yours is ironic, and all is forgiven. Waiting.)

    3. These e-mails said something fishy. Therefore there’s nothing going to happen to the Hatteras Light. Oh wait! The park service MOVED the Hatteras light! That’s why it’s falling down! I mean, not any more, but it’s clear that the Park Service’s decision to move the light was a politically motivated attempt to make the ends not only justify the means, but actually change the past! Those ba5tards!

    4. Name X Character Assassination x constant = truth (see Jonathan’s post above)

    5. Jesus! (at least four current republican candidates for president.)

    As for the list of assertions, sure; it’s a general or generic criticism. I’ll stay generic: there are 260,000 specific sea ice observations compiled in one average calendar month by one satellite program that is monitoring northern hemisphere sea ice. That’s one process among many for monitoring sea ice. That’s just sea ice, measured just one way. There are several other methods for monitoring sea ice; there are hundreds of other methods for measuring temperature trends in the arctic. There are dozens of ancillary observations that may or may not relate, such as observed wildlife habits, migration windows, etc. There are hundreds of related or subordinate components of the evaluation of those observations, and of course there are the politically exposed components, those that result in what we call “conclusions.” After that one might place the entirely political class of ideas, which you skipped to above for some reason, attempting to derail the argument by squatting on the most inflammatory possible component and then implying that the worst-case public policy solution impeaches the scientist who works with the sea-ice measurements or the one who programmed the computers that manage the satellite that gathered them.

    That’s one line of enquiry on the thesis that the world is warming. That line of enquiry has generated data, which has generated conclusions, which are rather conclusive enough to persuade the vast majority of informed and educated persons that the globe is warming well beyond what might be expected of the normal temperature trends, and faster by a factor of about 20, and hotter by a factor of 1-2. Etc. And please, by all means bring out the Oregon petition, which is one bit of climate debate where my math is sufficient to illustrate the depraved stupidity of the wingnut position, since only arithmetic is required.

    Your call for specificity is, of course, preamble to handwaving away a few outlier specifics and demanding binding generality, which will lead to your demand for particular cases since these persuasive generalities are obviously the result of a corrupt and monolithic science establishment. (never mind that scientists as a group are the least monolithical people ever invented and are far too preoccupied with abstractions to manage the simplest conspiracies, and would if they were so venal as you imagine be much more prone to attempt the refutation than to add one more to the thousands of corroborations already in the literature. Two researchers I know abandoned climate research because it was so boring, because the odds of an interesting finding were so slim–they walked away from what you laughably consider “lucrative” lines of research for the simple reason that you can’t distinguish yourself with corroborating or duplicating.) Then you’ll demand the scientists proffer the unprofferable, such as a block of raw data so you can run some science of your own. It’s not unreasonable to release data to researchers of comparable caliber and integrity, by the way. Not seeing too much of that from the same crew who thought it their ethical duty to wring tortured evidenceless conspiracy theories and misconduct charges from stolen e-mails they were simply too stupid to understand. Get specific–give us the data! We are scientists, too! you say, and prove it by failing to object when such distinguished scientists as the wingnut, Jesus-fixes-everything Attorney General of Virginia uses the Commonwealth’s judiciary and security apparatus and an utterly dishonest legal fig leaf to harass a scientist on data and calculations? Sure, I’ll cooperate there–how specific do you want? What? No longer interested in the data, are we? realized that we’d have to do science with it, and that is, like, all mathy and hard and totally takes ten years? Now you want to argue public policy? I have no opinion on public policy.

    Sea Ice

  15. You continue to argue by assertion. Listing researchers’ names and stating that evidence is undeniable is not scientific or even logical argument, it’s opinion. You are of course entitled to your opinions but I don’t find them convincing, nor do your wordiness and condescension help your case.

  16. ice9, you’ve done a great job of presenting the case for the anti-science Left. (or the Trofim Lysenko/Intelligent Design school of “science,” whichever you prefer.) To address a couple of points you seem to be unaware of:

    1. Go to any CAGW web site. Most will have multiple post/articles discussing the “proxies” used. In real science, a proxy is only legitimately used when direct observation and measurement are not possible. Thus, for example, using the Vostok Core sample to measure previous CO2 atmospheric content is direct data, using bristle cone pines to estimate temperatures in previous years is proxy data, and discarding data from Virginia’s Blue Ridge Mountains in favor of “proxy” data from the coastal plain is deceptive, shoddy science, and an indicator of fraud. Unfortunately, the CAGW community is awash in instances of the latter. Which brings us to:

    2. Legitimate theories in science account for all the data. They do not select data that proves the author’s point of view and discard the rest. There are serious problems across the board with the proxy data sets. Taking the most well known, the bristle cone rings form Siberia, a. they discarded the overwhelming majority of the data, and b. there is no explanation of how they determined that temperature, or length of growing season, was the limiting growth factor for any given year.

    3. Along the lines of data, there is no explanation provided for the “adjustments” made to the raw data from various sites, for example Point Darwin in Australia. Oddly enough, everywhere this has been noticed, the adjustments all have an upward slope, with temperatures prior to some date being downward, and those after being upward (the date varies, but is usually in the mid 20th century). Most people who understand science and mathematics find this troubling (including some who believe in CAGW).

    4. Models are mathematical descriptions of something. e.g. atmospheric transport, shear at the surface boundary layer, atmospheric mixing, cloud formations, etc. Simulations are a collection of interacting models that run over time. They do not produce data. There is no debate on this, although there are two sides. People with a sound grasp of the utility of models and simulation to test their understanding of the real world know this. Incompetents and frauds dispute it. Legitimate models and simulations have been subjected to a process of Independent Validation and Verification. When they are used outside the area for which they have been subjected to IV&V, they are only as useful as the availability of data they can be checked against, including any data that might show they are giving inaccurate result. The CAGW community fails dismally in this area, to such a great extent that it is impossible to even be generous and attribute the failing to incompetence.

    5. The animosity toward Dr. Spencer and Dr. Christy is due to the fact that the satellite data they are responsible for shows that the warming is not what the CAGW community “predicts” with their “models.”

    I could go on, but these should do. For the future, if you want to argue about science, start by learning how it works. The natural science survey course you took on the way to a liberal arts degree is not cutting it. But if you have a job in any scientific field, you should probably consider a career change.

  17. The Left, in general, is happy with the results of the Wikileaks exposure of private communications.
    Where is their enthusiasm for the exposure of all the emails and model ‘adjustment comments’ of the AGW crowd?
    When the CRWU emails were released, the AGW crowd was aghast that privacy was invaded. Where is their indignation at Wikileaks? Where is their indignation at the ‘adjustments’ to data without explanation? Where is their indignation at the location of data gathering equipment? Where is their indignation at discarding data which does not match their expected results?
    I read of ‘arm waving’ and ‘peer review’ and all sorts of esoteric verbiage which in actuality is just obfuscation used to distract the casual observer.
    We are not ‘scientists’. Fine, then science must refrain from using ‘consensus’ instead of provable theorems with measurable data. What is the exact prediction of the ‘change in average(crap number) temperature of the earth’ [again a crap number] related to the PPM of CO2 in the atmosphere? Where is the formula? What are the co-efficients? What are the variables?
    This is not science, this is innumeracy taken to the extreme.
    The value of determining the ‘average temperature of the earth’ over a year-long period approaches zero. That is one number, just one, that represents all the clouds, sun, storms, emissions, chemicals, fires, rainstorms and a whole lotta other things and it cannot possibly be reduced to a single number, with any possibility of a single digit to the right of the decimal point.
    Ridiculous on its face.
    tom

Comments are closed.