A real decision, worked the way BenchForge teaches: build the test so the score measures skill, not a trick — then read the hard cases, not just the average. You decide; the consequence is realistic; then reflect.
A real case · you decide
Help Dev design a test that can’t be fooled
Dev — a 17-year-old running a science-fair comparison of two study apps
The idea in play: design the evaluation so the score measures real skill, not a trick.
Dev wants to know which of two study apps actually helps students more. Both apps come with a big set of practice questions. Dev has to decide how to test them — and the test design will quietly decide the result.
It’s tempting to grade each app on the very practice questions it shipped with — the numbers come out high and clean.
But an app can look great on questions it has already shown its students. The honest question is how students do on problems the app never touched.
How should Dev score each app so the result reflects real learning?
A test built from data the app already saw measures recall, not skill — hold out fresh questions so the score means something.
On the fresh questions the two apps tie on average. How should Dev break the tie?
When averages tie, stratify: the hardest cases are usually where a real difference hides.
Dev tests both apps on fresh questions, then reads the hardest band instead of the average. The write-up doesn’t just name a winner — it explains why the test is trustworthy, which is the actual science-fair project.
