The engineering behind FilterForge
Every large game with a chat window has to decide what its filter flags. FilterForge puts you in the trust-and-safety engineer's chair: you set the rules, then measure them the way real teams do — not by a single "accuracy" number, but by the two errors that hurt different people.
The confusion matrix
Every message is either harmful or safe, and your filter either flags it or doesn't. That's four outcomes: a true positive (harmful, caught), a false positive (safe, wrongly flagged), a false negative (harmful, missed), and a true negative (safe, left alone). Two summary numbers matter most:
- Precision — of everything you flagged, how much was truly harmful.
- Recall — of all the real harm, how much you actually caught.
The two error costs
A false negative means a lure got through to a child. A false positive means a child talking to a real cousin was silenced. Both are real harms to real people — and they pull in opposite directions. Lower the threshold and you catch more lures but silence more friends; raise it and you protect friendly talk but miss more lures.
Why there's no perfect line
Harmful and safe messages don't sit in two tidy piles — their risk scores overlap. That's why no threshold ever reaches perfect precision AND perfect recall at once. The engineering question isn't "can we be perfect?" but "which error can we least afford, and who pays for each one?"
A real-world caution (honest yield)
Real filters make both kinds of error too, and the people who build them choose which one to accept. FilterForge uses a fully synthetic, non-graphic test set to build this intuition; it is not a real moderation tool and models no real classifier's true performance.
Where this idea comes from
Blackstone’s line
In 1769 the English judge William Blackstone wrote that “it is better that ten guilty persons escape, than that one innocent suffer.” A court sets its threshold high (proof beyond reasonable doubt) because, for a court, a false alarm is the worse mistake.
One thing to hold: A smoke alarm on a kitchen ceiling.
Why it matters: A smoke alarm leans the other way: it would rather go off for burnt toast than sleep through a fire, because a missed fire costs far more. A child-safety filter leans that way too: a missed lure costs more than a friendly message held for a person to check, so it sets a low threshold and puts a human after it.
What we don’t know: Where to put the line is never settled by maths alone. Too many false alarms have their own cost: people take the batteries out of a smoke alarm that goes off every time they cook, and newer editions of the US smoke-alarm standard (UL 217) add a test meant to cut those alarms.
Dates:
- 1769 Blackstone writes his ten-to-one line.
- 2015 The US smoke-alarm standard adds a test against cooking false alarms.