Shannon Love

It occurs to me that many of those reading my criticism of the Lancet Iraqi-mortality study don’t know what cluster sampling is or what types of failure it is prone to..

Cluster Sampling works like this:

Say you have 100 balls colored either black or white in an unknown ratio. Now, in a traditional random sample, you would put all the balls in one container and mix them well, blindly draw out a sample, usually around 10 balls, and then use statistics to tell you the likely ratio of all the balls.

In cluster sampling, you would divide the balls up evenly into ten different containers. Then you would select one container at random and dump out all of its balls and determine their color ratio. At first glance it works exactly like a random sample and in some cases it actually does.

Take these two extreme cases. In case A, the balls are thoroughly mixed before being put into the individual containers. In this case, counting all the balls in one container is the same as drawing a blind sample of ten from one container holding all the balls. In Case B, you start filling up all the containers with one color until you run out, and then you switch to another color. In that case, counting the balls in one container will give a wildly inaccurate answer EVERY time. For example, if you have a fifty-fifty ratio of black to white, you would have five containers with all black, and five with all white. Choosing any one container would give a ratio of either 100% black or 100% white.

So we have a spectrum where, at one end, cluster sampling works just as well as random sampling, and at the other end it fails every time.

Read more

A lot of sources seem to be pointing to towards a Coalition attack against the insurgent strongholds of Falujah and Ramadi sometime very soon. Many observers and commentators have wondered why it has taken so long for the Coalition and the provisional government to act .

I think they are waiting on the results of the U.S. presidential election.

Read more

I am embarrassed to say that when reading the infamous Lancet Study for my previous post, I was so stuck by the idiocy of using cluster sampling and self-reporting in a population (the Sunni) who have a strong motive to exaggerate that I just flat ignored the actual resulting statistics. Since I knew the methodology was crap I knew the numbers were crap and I didn’t look any farther.

Commentator JohnChris and Fred Kaplan over at Slate (via Instapundit) both pointed out that the confidence interval on the studies results, even with Faluja excluded, is so broad to be utterly useless.

Kaplan nails it so I will except a bit:

“Readers who are accustomed to perusing statistical documents know what the set of numbers in the parentheses means. For the other 99.9 percent of you, I’ll spell it out in plain English—which, disturbingly, the study never does. It means that the authors are 95 percent confident that the war-caused deaths totaled some number between 8,000 and 194,000. (The number cited in plain language—98,000—is roughly at the halfway point in this absurdly vast range.)

Imagine reading a poll reporting that George W. Bush will win somewhere between 4 percent and 96 percent of the votes in this Tuesday’s election. You would say that this is a useless poll and that something must have gone terribly wrong with the sampling. The same is true of the Lancet article: It’s a useless study; something went terribly wrong with the sampling.”

Of course, we know what went wrong with the sampling. The study’s basic design was flawed for examining a phenomenon know a priori to be highly asymmetrical.

This raises the obvious question: How did such a seriously flawed study get published in a prestigious (Lancet is the British equivalent of the New England Journal of Medicine) medical journal? The only possible explanation is political bias of the authors, the peer reviewers and the publisher.

Read more

Via The Command Post comes this study published in Lancet (free reg) which purports that 100,000 Iraqi have died from violence, most of it caused by Coalition air strikes, since the invasion of Iraq. Needless to say, this study will become an article of faith in certain circles but the study is obviously bogus on its face.

First, even without reading the study, alarm bells should go off. The study purports to show civilian casualties 5 to 6 times higher than any other reputable source. Most other sources put total combined civilian and military deaths from all causes at between 15,000 to 20,000. The Lancet study is a degree of magnitude higher. Why the difference?

Moreover, just rough calculations should call the figure into doubt. 100,000 deaths over roughly a year and a half equates to 183 deaths per day. Seen anything like that on the news? With that many people dying from air strikes every day we would expect to have at least one or two incidents where several hundred or even thousands of people died. Heard of anything like that? In fact, heard of any air strikes at all where more than a couple of dozen people died total?

Where did this suspicious number come from? Bad methodology.

Read more