It occurs to me that many of those reading my criticism of the Lancet Iraqi-mortality study don’t know what cluster sampling is or what types of failure it is prone to..
Cluster Sampling works like this:
Say you have 100 balls colored either black or white in an unknown ratio. Now, in a traditional random sample, you would put all the balls in one container and mix them well, blindly draw out a sample, usually around 10 balls, and then use statistics to tell you the likely ratio of all the balls.
In cluster sampling, you would divide the balls up evenly into ten different containers. Then you would select one container at random and dump out all of its balls and determine their color ratio. At first glance it works exactly like a random sample and in some cases it actually does.
Take these two extreme cases. In case A, the balls are thoroughly mixed before being put into the individual containers. In this case, counting all the balls in one container is the same as drawing a blind sample of ten from one container holding all the balls. In Case B, you start filling up all the containers with one color until you run out, and then you switch to another color. In that case, counting the balls in one container will give a wildly inaccurate answer EVERY time. For example, if you have a fifty-fifty ratio of black to white, you would have five containers with all black, and five with all white. Choosing any one container would give a ratio of either 100% black or 100% white.
So we have a spectrum where, at one end, cluster sampling works just as well as random sampling, and at the other end it fails every time.