Bigsby

That spring, a pair of mice had a new solitaire rule they were sure was brilliant. To test it, they dealt five games. The new rule won four. "Eighty percent!" they squeaked, already drafting an announcement for the whole valley. Bigsby read their notes, nodded politely, and asked them for one small favor. Would they run the very same rule again, on five different deals?

They did. This time it won two.

"Forty percent?" said the mice, staring at each other. "But it was eighty a minute ago. Which one is real?"

"Maybe neither," said Bigsby. "With only five deals, luck is loud. One easy deal lands in your pile, or one brutal one, and your answer swings a long way. Suppose your rule truly wins about half the time. Five deals could easily hand you four wins, or one, or five, just from which deals happened to come up. Five deals can't tell a great rule from an ordinary one that got lucky."

The lab's screen can show exactly how much the data can and cannot say. After each test it draws a bar called an interval, the stretch of true win rates that the results can honestly support. When the mice looked at the bar for their five-deal test, it stretched nearly across the whole screen, from barely-helps to helps-enormously. Bigsby leaned over and tapped the glass. "That's not a finding," he said kindly. "That's fog. There's a hill out there somewhere, the true answer, and you're trying to see its shape through a thick grey morning. You can't. Not yet."

"Then how do we see it?" asked the smaller mouse.

Bigsby lifted his tall stack and set it on the table with a soft, heavy thump. "We deal more."

They ran the rule on fifty deals. The bar pulled in from both ends, a good long way, and the hill began to show a faint outline. They ran it on two hundred, and the bar shrank again, narrower and narrower, until it sat in a tight little band. Now they could actually say something: the rule helped, modestly but really, and they could say it without crossing their fingers.

But the mice noticed something odd along the way, and to their credit, they said it out loud. "Going from five deals to fifty made the bar much narrower," said the smaller one. "Going from fifty to two hundred narrowed it much less, even though that was a hundred and fifty more deals. Did the extra deals stop working?"

"They didn't stop," said Bigsby, and his eyes crinkled, because this was his favorite part. "They just each did less. The first handful of deals clears away the thickest fog. After that, every new deal is thinning fog that's already thin. Here is the rule of thumb scientists use. To cut the width of your bar in half, you don't need twice as many deals. You need about four times as many. Half again? Four times again. The answer sharpens with the square root of how much you test, not in a straight line."

The mice worked it out on a scrap of paper. Five deals to twenty would halve the width. Twenty to eighty would halve it again. Eighty to three hundred and twenty, again. Each halving cost four times the work of the last.

"That's why you can't just toss in a handful of extra deals and call something certain," Bigsby went on. "A little more testing buys you a little more sharpness, and it keeps getting more expensive. A really sharp answer takes a really big stack. There's no shortcut through it. You just carry the stack."