Short or reproducing it, how can one judge the likely accuracy of a study?
Statistics won’t help. Statistics only tell one the odds the results spring from sheer chance, not whether your original measurements were valid in the first place. You get the same statistics from the same data set whether the data represent colored ping-pong balls, car wrecks or the lengths of salamander penises.
About the only way to calibrate the study is to see how it measures the same phenomenon that other studies measured. If the study’s methodology returns results consistent with other studies for one measurement, then we can be more confident that its other measurements are accurate.
The Johns Hopkins funded study of Iraqi mortality before and after the war (published with much media attention in The Lancet) has many critics and defenders. Is there any means of judging the study’s likely accuracy without reproducing it?
I think there is.