I’ll check into it.

http://scienceblogs.com/deltoid/2007/08/robert_chung_on_david_kane.php#commentsArea

If so, you might wish to reconsider and/or restate your position.

“I will send you the next draft of the paper so that you (and others) can see for yourself. That draft will be non-trivially different from this one and the major cause of the differences will be these comments”

I have no idea what your talking about.

If you are repeating that odd little conception that the inclusion of the Falluja cluster makes the study more accurate then you simply do not under statistics.

Statistics is all about extrapolating from small representative samples of a much larger population. To have any hope of doing that accurately, statistics must judge the likelihood that any particular sample represents a part of the larger population. It does that by seeing how far the sample diverges from the other samples. This is called variance. The combination of the variances of each sample tells us the variance of the entire set of samples. This in turn allows us to predict how closely the collective samples represent the larger population.

We have learned from both mathematics and practical experience that samples raise variance in all directions. A sample much smaller than the others raises the collective variance to a higher range as well as the lower. A sample much higher lowers the variance higher as well as lower. (This a simplified a little).

The problem with Falluja is that it is a massive outlier. It has over two times the deaths by violence as all the other 32 cluster combined do (52 vs 21). The studies authors themselves call it a significant outlier. Had they then excluded it from their main findings they would have been on solid ground. The study however, includes the Falluja data in its main finding. If it does not, it does not include the data which supports its main finding.

In any case, Kane’s argument is that the confidence interval for the change in mortality which the paper definitely states includes the Falluja cluster, cannot have the confidence interval that the author’s state.

The Roberts team made two estimates, one including the bus, and one excluding the bus, and concentrated on the one excluding the bus; then concluded that even if you exclude the bus the average occupancy on weekends went up

No, they did not. The main finding in the 2004 contains information that, based on the contents of the study can only be true if Falluja is included. That has been my chief all along. Almost every paragraph in the study depends on data from the Falluja cluster. Read it yourself.

I had thought you were challenging the study’s body count, which, in final form, was more than 600,000, not 100,000.

You’ve got the wrong study. The 2004 Study found 100,000 dead, the 2006 study found 600,000+ dead. Both Kane and I are talking about the 2004 study.

I do remember that you wrote:

“Even the most casual student of military history or, indeed, just a curious person with access to Google, should instantly know that the 250,000 figure would be far too high based on the direct observation of facts on the ground. You can’t kill that many people without leaving massive physical evidence. There is absolutely no precedent for killing that high a percentage of the population with air power (or even ground forces) but leaving so few clues that the information could only be teased out by an epidemiological study.”

But I see that you’ve changed your position now, which is a good, honorable thing.

So we are clear, Shannon, Falluja was EXCLUDED in the body count

I totally, complete, enthusiastically, joyously,consummately, entirely, fully, thoroughly, utterly, wholly, perfectly,altogether, without doubt, exhaustively, you go girl, absolutely agree with you that Falluja was not included in the widely quoted 100,000 excess deaths.

That has rather been one of my chief complaints since day one.

Again, here is the Interpretation paragraph from the abstract:

Making conservative assumptions, we think that about 100000 excess deaths, or more have happened
since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces
accounted for most violent deaths. We have shown that collection of public-health information is possible even
during periods of extreme violence. Our results need further verification and should lead to changes to reduce non-
combatant deaths from air strikes.

Now, can you tell me (1) if all the information in the paragraph comes from excluding Falluja, (2) if Falluja data is included what is the author’s stated rational for doing so, and (3) is the scientific or mathematical justification given valid in you opinion.

Your critique has, from the beginning, been based on the assertion that the study overestimates the body count.

No, my critique from the beginning has been that the paper, but not the actual data from the study, mis-reported the study’s finding about the percentage of deaths caused by violence and percentage of violent deaths attributed to the Coalition. I am perfectly fine with total EXCESS deaths of 100,000 as long as the authors are truthful about how the study’s data shows they died.

100,000 deaths from statistically earlier deaths of the elderly, increased illness, increased accidents. increased crime, insurgent attacks and coalition attacks seems perfectly reasonable to me. In fact, only a slightly lower number can be inferred from other sources such as the Iraqi ministry of health or UN studies.

I should point out that Kane’s critique has nothing to do with the body count. Instead, he is examine the variance in the papers stated increase in the risk of death.

The most important result from L1 is the first sentence of the Findings sec-
tion.2

“The risk of death was estimated to be 2.5-fold (95% CI 1.6 – 4.2) higher after the invasion when compared with the pre-invasion periods.”

Unfortunately, if the other results presented in L1 are correct, this con-
fidence interval is wrong. It is too narrow, especially at the lower end. The
Lancet authors cannot reject the null hypothesis that mortality in Iraq is
unchanged.3

That risk of death estimate definitely does include the Falluja cluster:

FindingsThe risk of death was estimated to be 2·5-fold (95% CI 1·6–4·2) higher after the invasion when compared
with the preinvasion period. Two-thirds of all violent deaths were reported in one cluster in the city of Falluja. If we
exclude the Falluja data, the risk of death is 1·5-fold (1·1–2·3) higher after the invasion.

The math debate centers around whether the researches used a symmetrical variance distribution (i.e the chance that an error was equally likely to be higher or lower) or an asymmetrical variance (i.e. the chance that an error would be more likely to be either higher or lower).

Kane assumed that the variance was symmetrical whereas his critics believe the variance is asymmetrical since the researches used bootstrapping (using the distribution of the real data to set the distribution of the variance) to get the reported results. Since data including Falluja is massively skewed upward, then the distribution of variance must be skewed upward as well which means that the variance could not drop below zero.

Kane might be wrong in his assumption. It is interesting that researches reported their bootstrap results but not the non-bootstrapped ones. This raises the possibility that the non-boostrapped results due support the null hypothesis as Kane says.

Interestingly, that hasn’t even slowed down the “Lancet proved wrong” meme from spreading like measles across the right-wing blogosphere.

I haven’t the math skills to understand some of the more arcane arguments about the study’s veracity, but this much seems clear: Shannon either misunderstands or is deliberately misrepresenting whether Fallujah was included in the study. After saying it’s inclusion is problematic, he later backtracks and tries out the theory that it was included in some places and not in others.

“They use the Falluja excluded set to provide a plausible number of excess deaths but then switch to the Falluja included set to provide a breakdown of the cause of death and the nature of the victims.”

So we are clear, Shannon, Falluja was EXCLUDED in the body count. Your critique has, from the beginning, been based on the assertion that the study overestimates the body count. Your latest canard about cause of death and nature of victims shouldn’t distract anyone from the fact that you got the most fundamental fact wrong.

Neither of those positions squares with what Kane has to say. I would quote from Kane’s paper, but it’s curiously become unavailable online.

Shannon can easily clear this up by quoting the parts of Kane’s work where Kane makes the claim that INCLUDING Falluja distorts the body count, which is what Shannon initially claimed.

The best way I can see for Shannon to extricate himself here is to admit that he jumped the gun on “vindication,” and that crowing prematurely reeks of political motives and subjectivity. An observer concerned with preserving scientific method and falsifiability could never make such a boast.

In general, the fevered responses to the Lancet/Johns Hopkins Iraq death estimates have betrayed a certain lack of strategic political understanding.

By turning what could have been an objective mathematical dispute into a poltical rallying cry, conservative ideologues undermine a cornerstone of their Iraq position, which is that a large number of civilian deaths is acceptable because the U.S. doesn’t target civilians and because those deaths serve a greater cause that will, in the long run, prevent ongoing bloodshed.

By accusing liberals of using the study to boost their case, conservatives unwittingly confirm that even they believe the massive civilian body count in Iraq is persuasive evidence that the invasion is a failure and/or was a mistake.

Not that I think the rapid spread of the phony meme that Kane has debunked the Lancet is a good thing. It does create a kind of noise that will allow some committed war supporters to remain in denial about its effects. But that is mitigating, or even more than offset, by unwittingly underscoring the relevance of civilian deaths, even as the U.S. military shamelessly “doesn’t do civilian body counts.”

You’re probably referring to Tim Lambert, a computer professor in Australia and more or less professional troll at:
http://scienceblogs.com/deltoid/

When I challenged the study data right after it came out he ripped me and I shot back, the result of which is that he has become obsessed with me and has a special “Fumento” category on his website. It is sadly one of the weaknesses of the blogosphere that those with the shortest working hours by definition have the most hours to blab their faces off on things they know absolute nothing about and to pursue vendettas.

That is because they did not reach that conclusion – Kane did. My comment was directed at your original statement that “… the confidence interval dips below zero, which is a big no-no.” You were wrong.

Kane is attempting to infer the CI for change in mortality from the CI given for pre and invasion mortality.

Using what assumptions about distribution?

The paper gives no indication that they used anything other than a normal distribution.

From the Deltoid comment thread already referenced:

David, what are you going on about here talking about the authors assuming normal distributions? A quick read back of Roberts et al (2004) reveals, at the bottom of p3, that

“As a check, we also used bootstrapping to obtain a non-parametric confidence interval under the assumption that the clusters were exchangeable. The confidence intervals reported are those obtained by bootstrapping. The numbers of excess deaths (attributable rates) were estimated by the same method, using linear rather than log-linear regression.”

So basically your analysis is completely tangential. Roberts et al got their confidence intervals by bootstrapping from the empirical distribution of their data. The empirical distribution of the data wasn’t normal and it wasn’t nearly normal.

There is insufficient reason to reject the null hypothesis” is a perfectly acceptable result from a study, which is what Kane is stating is the case

It is acceptable if you actually state that. Roberts et al did not. They have kept the confidence interval of the data set with the Falluja-cluster included secret. They only published the CI for relative risk with Falluja included.

Kane is attempting to infer the CI for change in mortality from the CI given for pre and invasion mortality. He find that 1.5 CI runs from -160,000 to +659,000. That translate to a 10% chance that mortality either improved or stayed the same.

This is important in statistic because it gives you an estimate of the data’s overall reliability. This number means that if we duplicated the study numerous times, 1 in every 10 studies would find that mortality had improved which is very unlikely to true. further if we duplicated the study many times the 9 out of 10 positive results would vary significantly from one another. One study would say 100,000 excess deaths the next would claim 400,000. Obviously, such results are useless.

Which is why the outlier of Fallujah was not included.

Yes it was. The main finding is wholly dependent on the Fallujah cluster. I can understand how you might not see that. The authors did everything possible to obscure that fact. Very dishonest.

Think for a moment – you’re in the position of saying that without including war zones, the sample shows a better than 95% chance that a lot of extra people died – but that if you include the war zones there’s a chance nobody extra died because of the war.

Completely wrong. The confidence interval is not a statement about how many people died but rather a statement about the chance that the study accurately measured any particular change in mortality rates within the CI range. Adding Falluja doesn’t statistically lower the chance that more people died, instead, it destroys the accuracy of the actual number of excess deaths. Think about it, there is a 1 in 10 chance that, no matter how many actual excess deaths occurred, the study will report zero or negative (improve mortality).

The method is flawed if you assume a unimodal distribution – which the authors didn’t.

The paper gives no indication that they used anything other than a normal distribution. Given the stated design of the study i.e. that clusters were interchangeable, there was no reason for doing so.

That doesn’t make sense. The reason it doesn’t make sense is because of a statistical artifact that occurs if and only if you treat the full sample – war zones and non-war zones – as a unimodal distribution. The authors didn’t, as the people in that thread spent much time trying to explain to Kane. Again, examine this.

I not sure of the provenance of that chart but the bottom one shows an asymmetric distribution with a rather big chunk of the curve south of zero. That would tend to support Kane although by other means.

Kane’s entire analysis is based on the exclusion of Falluja

You are hallucinating. The entire point of Kane’s article is to try to reconstruct the missing confidence interval for data set with the Falluja cluster included.

The Falluja cluster was NOT included in them.

The Falluja cluster is most definitely included in the paper’s main finding. The sentence in the Interpretation that says:”Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.” is only true with the Falluja cluster included.

Les Roberts et al VERY dishonestly switch back and forth between two data sets throughout the article without any indication they have done so. One dataset excludes the Falluja cluster and the other includes it. They use the Falluja excluded set to provide a plausible number of excess deaths but then switch to the Falluja included set to provide a breakdown of the cause of death and the nature of the victims.

That is why I claim this is a political hatchet job. The authors cherry picked their data to create exactly the story they wanted to tell. If anyone else had done this in any other context everyone would be screaming bloody murder about.

You can’t get “Eighty-four percent” of violent deaths caused by Coalition, or “95 percent due to air strikes and artillery” without including Fallujah.

You also can’t get a “58-fold increase in death from violence” without including Fallujah.

etc. etc.

Work it out with the data if you don’t believe me.

What they did is not use it in the “100,000″, but still used it indirectly there to create the impression that their study had set a floor of 100,000 on excess deaths (which is part of what Kane is on about).
The authors rarely said when they were or weren’t using it, which is why you don’t know this.

The Falluja cluster was NOT included in them. This is exactly what Kane’s analysis is based on–the fact that Falluja was NOT included.

The way this story has spread across blogs reminds me of what Mark Twain said: “A lie circles the earth before the truth even puts its shoes on.”