From the record
Real cases, written from the published record. Predict first; the record and its sources come after.
1951 · Britain
The doctors who smoked
You cannot assign people at random to smoke for thirty years. So in 1951 Richard Doll and Austin Bradford Hill wrote to every doctor in Britain, asked how much they smoked, and then followed who died of what.
About 40,000 doctors replied.
As the deaths came in, how did lung cancer relate to cigarettes smoked per day?
What happened. Lung-cancer deaths rose step by step with the number of cigarettes, and doctors who stopped smoking did better than those who kept going. Bradford Hill later listed what had made them sure: the strength of the link, the dose-response, the timing, the consistency across studies, and how plausible it was.
One of the most important causal findings of the century came from a survey, done carefully.
2015 · worldwide
The 100 replications
270 researchers took 100 findings published in three psychology journals and re-ran each study carefully, with the original authors' help where they could get it.
Almost all of the originals, 97 of the 100, had reported a statistically significant result.
How many came out significant again?
What happened. 36 percent did, and on average the effects were about half as large as first reported. Originals with weaker evidence (p-values close to the cut-off) and more surprising findings were the least likely to hold up.
What we don’t know. A failed replication does not prove the original was false; some differences in method or sample may matter.
Pause to think. Which would you have bet on before the re-runs, and why?