Important thought on New Orleans you should seriously consider…

Which is where we came in; any departure from the Platonic Form of the Epidemiological Study can be tricked up into sinister conspiracies to bias results by a suitably unprincipled “sound science” hack.

Skip, this is what the “design effects” included in the survey’s confidence intervals are there for. Do you have any specific reason why this methodology is not appropriate to this dataset, or are you just trying to claim that the researchers didn’t address cluster sampling issues when they did?

Furthermore, the effect that we are looking at here is a cohort study. While the cross-sectional variance is affected by the clustering, it is hard to see how the in-group variation of each individual cluster could be. For any of the clusters, it is very hard to argue that the post-invasion death rate is a draw from a process with the same expectation as the pre-invasion death rate.

Just repeating my point from another thread. A standard deviation of 25% of the mean across clusters is actually very bad news. Ideally, the standard deviation across clusters should be zero, with all variation occuring within clusters. Each cluster should be as close as possible to a smaller version of the entire population.

“And every single governorate except one saw a rising death rate.”

Not strictly true. Due to their adjustment of the clusters, 6 of the governorates were not sampled at all.

By they way, if you want to understand why statitics won’t rescue you from poor clustering read The Madness of Methods

This is actually incorrect. In fact, cluster sampling when the clusters are heterogeneous population is notorious for producing samples that are not representative of the population as a whole.

–pg. 3, bottom of col. 1
–bottom of Table 2
–pg. 4, bottom of col. 2
–pg. 6, middle of col. 1

The infant mortality figure that Roberts et al. produce is within a factor of three or less of that which can be supposed to be correct from other published sources. Shannon Love discusses this finding in a post subsequent to this one.

My apologies.

Are you claiming that the authors didn’t calculated the standard errors correctly, or that they didn’t include a design effects term? Because that would be untrue.

Are you, on the other hand, claiming that it would be perfectly normal to get a sample of this kind by chance if the Iraqi death rate had not increased substantially? Because that also would be untrue.

You might also note that one of the clusters *did* land in Sadr City. That cluster happened to have zero deaths in it. The authors did not strike it out as an outlier.

If you have been following the rightwing propaganda about the Lancet study, you will be glad to see that a few people are weighing in with considered opinions. The original wingnut that I criticized finally seemed to recognize the mistake I pointed o…

Thanks for checking out my comment. I’m really new to the commenting on political blogs thing (mostly I just hang out at my friends’ personal blogs.) Who knew this could be so consuming!

It is true that cluster sampling works best when each cluster is a microcosm of the whole population you’re trying to capture (see http://www.brainyencyclopedia.com/encyclopedia/c/cl/cluster_sampling_1.html for a mini primer) but often this is not the reality. They used cluster sampling because conducting a door-to-door simple random sample just wasn’t feasible. Iraq is a big place and not so easy to get around as we all know. However what they did do was try to minimize effects of outlier clusters (by reporting rates excluding Fallujah.) and being very clear about where their clusters were and what they found. On page 7 they mention how their cluster in Sadr City oddly reported no deaths. The hope is that these issues will balance out somewhat. But no one, not even the authors were claiming it’s perfect.

I agree with you that violence isn’t random. But I’d like to point out that their use of statistical tools was to try to show what role chance may have played. The authors are very clear that systematic error may be present.

The more widely-accepted a method or result is, the less-necessary multiple approaches are. But the more ‘wild and wooly’ a claim is, the more valuable internally consistent verification becomes.

Plainly, Roberts et al. has no such mechanism for their extrapolation of circa two dozen recorded violent deaths to a nationwide invasion death toll. But they could have included one. They could have used their sample to estimate the toll of a violent episodes in Iraq’s recent history that has some reliable range of casualty figures associated with it. Three such are:
–Military casualties of the Iran-Iraq war of the 1980s;
–The Ba’athist executions in suppressing the rebellions of the early 1990s;
–The elevated infant mortalities due to Ba’athist abuse of the Oil-for-Food program of the late 1990s.

IIRC, the first two have casualty figures in at least the high-hundreds-of-thousands, and the third led to tens of thousands of excess infant deaths. These magnitudes are thus comparable or greater than the civilian deaths caused by the invasion.

What confidence intervals would Roberts’ methods have returned for these episodes? Would they have been wide or narrow? Consistent with published casualty estimates, or grossly inflated? This information would have been very helpful in evaluating the application of the methods they use in the case under study here.

I can think of some possible practical reasons why Roberts et al. did not take this approach. I can also think of other, politically-motivated reasons for rejecting this more-careful approach. Were I a reviewer of this manuscript, it is one of the points I would certainly have required the authors to address.

Thanks for your cognizant input but Kaplan’s et al point still stands. The broader the confidence interval the sloppier the data.

As a another poster pointed out, the CI in the paper tells us that if we did 20 identical surveys, statistically 19 would give a casualty estimate anywhere between 8000-194000 while the 1 would be above or below that range.

All you really need to know about the paper though is that their “conservative” estimate of 51,000 violent deaths is statistically extrapolated from 21 unconfirmed reports of actual violent deaths. The only way this has a chance of being true is if the violence is symmetrically distributed across the whole of the 24.4 million Iraqi population but we know that this in absolutely not the case. Most Iraqi’s have seen no violence from the Coalition and large stretches of the country, especially in the eastern regions, have seen no serious fighting at all.

The researchers tossed out their Faluja findings because they were so obviously not representative but 7 of their clusters 33 clusters where in Baghdad so if only on of those clusters feel in the Sadar city or the like then they would have another outlier.

Violence isn’t random, especially in warfare. You can’t study it with statistical tools. Just because one particular neighborhood or region gets hammered tells you nothing about the likelihood that a similar but distant region will see violence.

I’m Just seeing this post now, right after I posted a long comment about confidence intervals on your blog entry from October 29th. Hope you have a chance to go back and check it out, thanks!

Chicago Boyz takes apart the Lancet study that Jon linked a few days ago indicating that 100,000 civilians died during the invasion of Iraq. The verdict: scientific malpractice.

VodkaPundit, whose wife waited 5 1/2 hours for early voting in an unco…

Yep, we choose our heros, and Scientific Materialism seems to be the poison of choice for these past hundred years. Like the journalists who are annoyed that they cannot set the political agenda, there are those who see the effects of scientists and scientism on philosophy and society and cannot resist the shock effect of extreme scenarios.

People with power will misuse it.

Matya no baka