The engineering behind FilterForge
Every large game with a chat window has to decide what its filter flags. FilterForge puts you in the trust-and-safety engineer's chair: you set the rules, then measure them the way real teams do — not by a single "accuracy" number, but by the two errors that hurt different people.
The confusion matrix
Every message is either harmful or safe, and your filter either flags it or doesn't. That's four outcomes: a true positive (harmful, caught), a false positive (safe, wrongly flagged), a false negative (harmful, missed), and a true negative (safe, left alone). Two summary numbers matter most:
- Precision — of everything you flagged, how much was truly harmful.
- Recall — of all the real harm, how much you actually caught.
The two error costs
A false negative means a lure got through to a child. A false positive means a child talking to a real cousin was silenced. Both are real harms to real people — and they pull in opposite directions. Lower the threshold and you catch more lures but silence more friends; raise it and you protect friendly talk but miss more lures.
Why there's no perfect line
Harmful and safe messages don't sit in two tidy piles — their risk scores overlap. That's why no threshold ever reaches perfect precision AND perfect recall at once. The engineering question isn't "can we be perfect?" but "which error can we least afford, and who pays for each one?"
A real-world caution (honest yield)
Age-detection is a famous example: a 2026 age-band system estimated ages with an average error of over a year — enough to mis-sort a 12-year-old and a 14-year-old. FilterForge uses a fully synthetic, non-graphic test set to build this intuition; it is not a real moderation tool and models no real classifier's true performance.