Think some more. This would be tantamount to assuming that 1 in 10 of the new drivers killed someone in a car in their first year of ownership. That takes quite some doing when you spend most of your time queuing for petrol.

The US military has about 150,000 soldiers in-country, about ten times the estimated size of the actual guerilla force (this is the standard counter-insurgency ratio). That is about the same number of Iraqi persons who, the study predicts, have received disabling wounds in this conflict. (Obviously the Iraqi medical facilities are inferior, thereby giving a higher mortality ratio.)

Why would it be surprising that the US would inflict one Iraqi military casualty per annum for every one soldier that is on active service in this theatre of operations? Given the kind of firepower at its disposal, the urban context of the battle and the popular nature of the insurgency it is surprising that the US has not inflicted more casualties. (I put this down to smart weaponry, careful targetting and discriminatory use of fire power.)

Or, to put it another way, I would not be at all surprised to see that the US killed ten times as many insurgents as it lost with serious wounds. ie 10,000 US WIA to 100,000 Iraqi KIA. The US military’s force amplifiers are easily an order of magnitude greater than the organic weapons used by the insurgents.

Boy, that seems really odd. Considering all the execution chambers, torture chambers et. al. found in Fallujah by the Coalition forces.

One wonders how those deaths could have failed to make the study. Methinks their statistical analysis wasn’t the only thing in here that was severely flawed.

Daniel Davies comments
on the current outbreak of Lancet denialism, including Shannon Love’s latest
effort w…

I suppose I fault the authors for this “confusion” more so than you do.

Heiko, this is something I think everyone ehre agress on, and nowhere in the Lancet study does the authors equalize war dead with excess dead. Critiquers and people using study for political statements often do it, but that is hardly the fault of the authors.

Traffic accidents are just not morally equivalent to soldiers shooting children carelessly.

And excess death for this conflict doesn’t have to be defined over such a short period as two to three years. The invasion will have effects for decades, and not just in Iraq, but worldwide.

Worldwide, Unicef claim that 11 million children die that wouldn’t have to. There’s a million traffic accidents, the death tolls from Aids and Malaria.

The true impact of the invasion lies in the longer term picture, and whether it contributes to a stable, democratic Mid East, that allows Asian nations to grow and substantial resources to be devoted to solving Africa’s problems.

Of course, not everything in the future depends on the outcome in Iraq, but the figure of excess deaths or people saved over the next few decades will likely absolutely dwarve the impact on the death rate in Iraq that’s been experienced so far.

Excess dead can disappear, or excess “lives saved”, can be countenanced by all the lives lost in a future civil war and chaos in the Mid East.

War dead stay war dead, a soldier who shoots a child cannot make that undone, no matter how much infant mortality falls.

The moral culpability for war dead is very different from excess death.

I would like to close with the role of accidents in the claimed excess death figure.

After thinking the matter through, I think that the importation of hundreds of thousands of second hand cars has the potential for 10 or 20,000 extra deaths.

Iraq has many young people, people in an age group known to tend towards risky driving. I believe many of those would have had access to a car for the first time over the last two years, so not only do we have more young drivers on the roads, but worse drivers with hardly any experience, who’d have to cope with much more crowded roads, and who’d be driving very old cars.

The Lancet study’s results would be consistent with an extra toll of over 20,000 from road traffic accidents. But should they be counted as “war dead”?

Thanks for the additional details regarding Garfield’s correspondence with you. However, from my perspective, it muddies the water further in the context of the Interpretation statement from the study. And yes, I think you’re correct that you didn’t explicitly commit to a 30,000 bombing death figure.

When Garfield states that

“‘Majority of deaths from the air’ was mistated as being based on non-falluja experience.”

is he referring to a misstatement in both his EPIC interview and the Interpretation statement form the paper, or just the former? Clearly, his last statement to you only refers to his own error at EPIC.

I can’t see how he can separate his error at EPIC from the statement in the paper, because they’re essentially saying the same thing, and in doing so, making the same error.

To put this another way, is this really Garfield’s faux pas, or do all the authors share it? I can’t reconcile the fact that Garfield’s error also seems to be identical to the error made by observors and commenters not involved in the study, and was based largely on the key statement from the study. I believe the 60,000/30,000 figures had actually been adopted at your site before we were aware of Garfield’s interview.

‘Majority of deaths from the air’ was mistated as being based on non-falluja experience. It is methodologically interesting and important to try to figure out what is going on with a representative but very small sample. On this point, I combined data from non-falluja area and the total mortality experience, including falluja. My mistake.

Thanks again for the kind words. I do in fact recall you expressing your disagreement with the use of extrapolated numbers, particularly attributions for individual death subsets. I may well be wrong to suggest you accepted the 30,000/60,000 figures.

That being said, the extrapolated numbers are to some extent simply another way of expressing the increase in the relative risk. In other words, criticism of the extrapolations is also criticism of the accuracy of the increase in the relative risk. The data used to calculate both are the same.

I won’t bother to repeat my argument from my original post, dealing with my perception of the volatility from study to study of certain key death subsets, partly beccause the argument has admittedly lost some of its punch in the wake of Garfield’s correction.

For the record, I have been in agreement with Tim at Deltoid over the need for additional studies. On one occasion, I suggested the best option might be an actual count for the Falluja cluster, accompanied by a broader cluster sampling for the rest of the country. However, now I’m of the belief (as I mentioned in my post at the ” labcoat ” thread here) that an actual count of all violent deaths is the best and most accurate approach, at least for purposes of determining the violent death component of the aggregate excess death figure. As I mentioned previously, such a process is nearing completion in the Former Yugoslavia, and is shattering the previously accepted death figure of 250,000.

I do have one disagreement with you, D Squared, in relation to your statement that

“I’d note that, of course, this is the benefit of a really good peer review like the Lancet study received; it weeds out loose and indefensible statements like this one, which is why there aren’t any in the actual paper.”

I believe the contentious ” Interpretation ” statement on the study’s page 1 must clearly be viewed as a ” loose and indefensible statement” when dealing with the portion of it that labels coalition air strikes as accounting for the majority of violent deaths, especially now that Garfield has retracted his near identical statement from the EPIC interview. I agree though, that this has no effect on the aggregate excess death figure or the risk ratio.

Garfield is AFAICT off base here; the data does not support robust estimates on the breakdown numbers, although it does on the overall risk ratios. My very first Lancet post said right up front that I don’t like extrapolated numbers and this is a good example of why; you can’t base extrapolated numbers on outliers, but you also can’t discard the information from clusters like Fallujah. I’d note that, of course, this is the benefit of a really good peer review like the Lancet study received; it weeds out loose and indefensible statements like this one, which is why there aren’t any in the actual paper.

On the other hand, in my second most recent Lancet post, I pointed out that it is a bit much for people to be banging on about this or that detail of the breakdown of the extrapolated number when they are a) visibly not calling for a more accurate, better-resourced study to be carried out b) visibly not condemning the coalition forces for “not doing body counts” and c) trying their level best to minimise or ignore the whole study and its conclusions, which are robust.

(neutral observers will see that there is a sort of travelling “Lancet study community” who swap comments on all sorts of different websites and remember all their past debates. You can join this community if you like; the only real membership criterion is that, because we’re a bit serious about the science, we don’t call people “scientific whores”)

Sorry for not replying sooner, had to get some sleep. Thanks for the compliment D Squared.

You’re both correct that the 30,000 bombing death estimate, ex-Falluja, does not specifically appear in the study text. This was arrived at from two sources. One is the EPIC (Education for Peace in Iraq) interview that study author Richard Garfield gave in November 2004. This is the interview that Tim is referring to when stating that he contacted Garfield, and Garfield is now on record that he was “ mistaken.”

I’ve provided the URL to the Garfield interview in my previous post. The article contains a bar graph apportioning the deaths from the 100,000 excess death estimate. 30,000 deaths are attributed to coalition bombing, and 57,600 for all violent deaths. Just below the graph, Garfield makes this statement, which he has now apparently corrected:

“In areas of Iraq, with the exception of the North, all had a rise in the mortality rate and most were due to violence. Real change was in deaths due to violence.[The majority of the 57,600 deaths due to violence was attributed to air assaults.]”

The second source for the 30,000 bombing death estimate comes from the study itself, and is contained in it’s “ money quote, “ from page one:

“Interpretation: Making conservative assumptions, we think that about 100,000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces accounted for most violent deaths.”

While there was some discussion at Deltoid that the authors may have also been lumping the Falluja deaths in when making this conclusion, it seemed to me that the consensus was this premise couldn’t be defended, especially after you yourself linked to the Garfield interview on December 15th. The fact that one of the study authors had pegged ex-Falluja bombing deaths at 30,000 put an end to any substantive argument that the authors were including Falluja.

I think we had all come to accept that the study’s “ Interpretation “statement simply couldn’t have been made so sloppily and ambiguously, given its importance to the study. Since the risk of violent death had risen 58 fold post-invasion, and based on the above statement, it was agreed at Deltoid (and elsewhere) that 58,000 of the 100,000 excess death estimate came from violence. For simplicity, the rounded figure of 60,000 was consistently referred to at Tim’s site. Again, based on this key statement from the study itself, that the majority of violent deaths were from coalition air strikes, the figure of 30,000 bombing deaths was adopted.

To be fair Tim, I think you should acknowledge that the 30,000/60,000 figures were adopted as accurate by yourself, myself and other regular commenters during the early Lancet threads at your site. D Squared, I believe you were also in the boat with the rest of us, but I can’t confirm that without retracing all of Tim’s very lengthy threads. I think I’m on firm ground when I say I don’t recall you arguing to the contrary at the time.

Just to set the record straight then Tim, I think it’s somewhat unfair to infer that I’ve been pushing inaccurate numbers in this area, of my own making. Here’s a quote from you on December 9th, from your “ Lott on the Lancet “ post, November 28th:

“Mike, thanks for getting the breakdown of the figures. The pre and post-invasion periods were of unequal lengths, so you would need to adjust for that when working out how many excess deaths there were. I think you get about 60% from violence (not counting Falluja).”

I’ll understand if you don’t want to reveal the contents of Garfield’s explanation, out of respect for his privacy. If on the other hand you could provide some details concerning his correspondence with you, I’d be very interested. I’m curious as to whether he provided any explanation on HOW he managed to make this error.

I have several concerns with the revelation that Garfield has retracted his figures and statement from the EPIC interview. Of course the most obvious one is, how could one of the study’s leading figures get something this crucial so wrong? I’m also concerned with the fact that other study authors such as Les Roberts and Gilbert Burnham reinforced in interviews the point that the 100,000 figure excluded Falluja. In my view, this very much calls into question their motivation for wording the key “Interpretation” statement the way they did. This also renews the controversy over why the authors failed to clarify in the study the breakdown and distribution of non-coalition caused violent deaths. Without clarification obtained from the authors outside the study, it was impossible to discount the Interpretation statement that most of the excess violent deaths in the 100,000 estimate were from air strikes. Now, we not only know this statement is incorrect, we also have one of the authors admitting it.

I’ve seen many heap blame on the media for the widespread misrepresentation of the 100,000 death estimate, while at the same time deflecting blame from the authors for this. Tim, I think your revelation concerning Garfield’s admission changes this dramatically, and in fact the authors deserve the lion’s share of the blame for this.

“I’m upfront about my lack of specialist statistical knowledge. Like many educated people, I often make informed judgements based on what I have learned, combined with common sense. That you—an expert in this area—see things differently is not altogether surprising. Tellingly, you say “common sense is almost always a poor guide to good statistical practice.” (point 3) As far as the fundamentals of statistical practice, my old stats prof would dissent. I suspect Richard Feynmann would, as well.”

Google on “Kahneman Tversky”. Not only do laypersons (non-statisticians) do HORRIBLY in judging situation that rely on probabilistic reasoning, even statisticians do horribly on it. Dsquared is right on and there is voluminous evidence to support his notion that “common sense” gets statistics wrong.

Love, it is common practice to round a number to indicate the number of significant digits in it. A number of 98,000 would suggest that the real number was between 97,500 and 98,500, which is misleading. Yes, you can specify the standard deviation or 95% confidence interval to give an exact description of the uncertainty, and that was what was done in the article, but to the wider audience that would just be confusing, and even worse, use it in a press release and you can be certain that some journalists will drop that qualifier and just say 98,000 dead. Rounding to 100,000 was the most honest thing to do.

What is your particular complaint about the handling of the outlier, and how do you think it doesn’t comply with the Kruskal guidelines?

Kruskal: “My own practice in this sort of situation is to carry out an analysis both with and without the suspect observations. If the broad conclusions of the two analyses are quite different, I should view any conclusions from the experiment with very great caution.”

AMac, 3/23 5:18pm: The claim of [S. Love’s fisking is] that Roberts misleads the Lancet’s generalist audience, conflating the IFC and EFC analayses to produce a desired result.

AMac, 3/24 11:59am: [I came to agree with S. Love’s charge that Roberts conflated IFC and EFC analyses.] My method was to print out S. Love’s fisking, [dsquared’s] rebuttal (3/23 2:47am), Disputo’s critique (3/23 6:43am), and then go through the Lancet paper.

dsquared, of course you disagreed with me, and will continue to do so. I am not expecting consensus. Interested readers can do as I did, and draw their own conclusions. Perhaps some will find your and Disputo’s points to be convincing.

What specific methodological flaws do you think this study has?

This refers to my point 5 of 3/23 6:36pm, reproduced in its entirety: While the study has other problems, I can’t evaluate their seriousness. But every study has problems.

Let me rephrase this minor point: I’ve focused my writing to where I thought I might advance the discussion. Critics have highlighed other potential problems. While every study has problems, elucidation of the facts and contexts of these other aspects of Roberts will depend on exchanges between those critics and Roberts’ defenders.

What errors do you think a layman would be likely to make as a result of the “misleading” summary?

(This response assumes no serious methodological flaws in the study.)

–That the study has clearly shown that, conservatively, at least about 100,000 excess deaths have happened since the 2003 invasion of Iraq. This Oh That Liberal Media website post gives seven examples of reporters who, indeed, took that as the take-home message.

–That the statement “after the invasion, violence as the primary cause of death” is supported by much stronger data than 21 violent deaths out of 90 total, or 73 violent deaths out of 142 total, if the excluded cluster is re-included. Recall Kruskal.

–That “Violent deaths were widespread, reported in 15 of 33 clusters, and were mainly attributed to coalition forces” is supported by much stronger data than 21 violent deaths in the non-outlier clusters, of which 9 were reported as due to coalition action. Yes, the sophisticated reader can glance up two paragraphs, sees 33, mentally subtract 1, and deduce that the sentence must refer to outlier-included statistics. Recall Kruskal.

–That “Violence accounted for most of the excess deaths [antecedent for most: “about 100,000”] and air strikes from coalition forces accounted for most violent deaths” is supported only if the Fallujah cluster excluded from the immediately-prior sentence is re-included for this statement. More than violating Kruskal’s precept, this grammatical sleight of hand invites misinterpretation.

As you say, you think the Summary is an accurate account of the evidence. Again, I am not expecting consensus. Again, interested readers will draw their own conclusions.

I don’t think that going further rounds will induce many long-distance thread-readers to change their opinions on these matters. Though I won’t post again on this, I will of course look forward to reading any response, likely disagreeing with much of it.

Thanks also for the overall politeness you have shown me on this thread, though this appreciation is blunted by the rather, um, vivid language you and your pals use at your own site in describing the mental faculties and personality traits of those of us who have commented negatively on Roberts. I confess the latest Crooked Timber thread has the flavor of one of Michael Kinsley’s gaffes, (paraphrasing), what happens when a politician says what he really means.

Regards, AMac

But comparing their stats to the Lancet’s is an apples and oranges game, and it’s probably wise to keep them out of the equation(s) here.

Someone asked for and someone else (dsquared, actually) provided the Lancet’s error bounds for their post-war mortality calculations with and without the Fallujah cluster, viz:

The mean and confidence interval for the post-invasion death rate including the Fallujah cluster is given on page 4, bottom right-hand corner. It’s 12.3, CI 1.4-23.2, design effect 29.3

Now one noteworthy number here is the “design effect” (DE) of 29·3. This is a measure of the reliability of a survey using cluster sampling (CS) when compared to a similar-sized survey using a simple random sample (SRS). Where the DE is 2, the effective sample size of a random CS survey when compared to an SRS with the same number of sampling units is halved.

So to drop the acronyms, a random cluster sample survey of 1000 households whose “design effect” is 2·0 will have the effective sample size – and reliability – of a simple random survey of 500 households where each was selected independently of all the others (you can guess why that isn’t done very often, and if you can’t, just think gasoline prices, travel time, wages, money).

And how does anyone know what the design effect of a particular survey will be? Well, you can’t work this out until you’ve collected at least some results, because the variation between clusters plays an important role in calculating it. Suffice it to say that the smaller the variation found to exist between clusters, the smaller the design effect, and the less a survey will suffer in comparison to an SRS, the (expensive) ideal.

Readers who’ve been paying attention will be able to work out for themselves the “effective sample size” of a cluster sample survey of 988 households whose design effect is 29·3. That the authors of the Lancet study decided to drop the Fallujah outlier is hardly surprising when you consider their discovery that:

If the Falluja cluster is excluded, the post-attack mortality is 7·9 per 1000 people per year (95% CI 5·6–10·2; design effect=2·0). [page 5, left columm, first paragraph]

Much better to base estimates on an effective sample size of 494 households than on a sample with an effective size of 34!

I leave it to fact-checker extraordinaire dsquared to issue all needful corrections, etc., regarding the above.

Hahahahaha. You can’t seriously mean this, can you? Of course it does. Down is not the same as up. You screwed up. Admit it.

More over, if they did calculate the figure as you suggest the researchers were incredibly sloppy. Remember this is a cluster study. Each cluster represents a unique area of Iraq calculated by the demographic software. If a cluster is removed, the remaining cluster must be recalculated to stretch over the area of the missing cluster.

This is, to put it bluntly, nonsense. I am shocked that you have the temerity to mention “this is a cluster study” when you have been caught out in a complete fabrication on the subject of cluster studies in the past (one which, I hate to repeat myself, you have yet to admit to). Do you really have so little respect for me that you expected me to fall for this?

Each cluster represents a unique area of Iraq calculated by the demographic software.

Which demographic software? Each cluster represents the thirty households closest to a random geographically selected location and is extrapolated to represent “3% of the population”. The study says, several times, that each cluster represents 3% of Iraq.

If a cluster is removed, the remaining cluster must be recalculated to stretch over the area of the missing cluster

This is just simply meaningless verbiage. It really doesn’t refer to anything. The transformation involved is just a scaling.

1. What your particular complaint is about the handling of the outlier, and how you think it doesn’t comply with the Kruskal guidelines

2. What specific methodological flaws you think this study has (if you’re endorsing some of Shannon’s, it would help one heck of a lot if you could provide links).

3. What errors you think a layman would be likely to make as a result of the “misleading” summary. I don’t think it is misleading; I think it’s very accurate as a summary of the evidence. Therefore, I don’t think it’s on to just say “this was misleading” without saying how it was misleading. I think I’ve dealt with all of Shannon’s points with respect to this; if you don’t agree, once more, links or excerpts would be very helpful, thank you.

“It says that the figure of 98,000 refers to the 97% of the country represented by clusters other than the Fallujah cluster. 98,000 divided by 0.97 is 101030, which rounds to “about 100,000?”

So they either rounded up from 98,000(2,000) or down from 101,030(-1,030). This in no way changes the validity of my criticism. I am pointing out that they rounded to the more marketable number of 100,000 instead of providing the actual number, whatever that was. There was no technical reason to round at all.

More over, if they did calculate the figure as you suggest the researchers were incredibly sloppy. Remember this is a cluster study. Each cluster represents a unique area of Iraq calculated by the demographic software. If a cluster is removed, the remaining cluster must be recalculated to stretch over the area of the missing cluster.

I suppose it is possible that they recalculated everything and ended up at exactly 100,000 but what are the odds of that?