Real cases
A case · you decide
Vale and the 99% filter
Vale — a mentor reviewing a teen team's synthetic chat-safety filter
The team's demo slide says the filter is 99% accurate. In their synthetic test set, 1 message in 100 is harmful.
Vale points out that a filter which flags nothing at all is also 99% accurate on that set. It misses every harmful message.
Two numbers tell the real story. Precision: of what was flagged, how much was really harmful? Recall: of all the real harm, how much was caught?
The filter flagged 50 messages; 32 of them were really harmful. The set held 40 harmful messages. What should the slide say?
The new slide shows 80% recall and 64% precision on the synthetic set. Those are lower numbers, and they're true.
Who made this?
“In reality, every technology reflects decisions made by designers, companies, and communities, and carries the values and power of those choices.”
- Who benefits from this technology?
- Whose needs are ignored?
A case · you decide
Thresh and Cousin Cora
Thresh & Cousin Cora — the filter's threshold dial, weighed against the false-alarm cost
Lowering the threshold catches more of the synthetic harmful messages. It also flags more ordinary family chats, like a kid messaging a real cousin.
Cousin Cora stands for the false-alarm cost: a real relationship wrongly silenced.
Every threshold setting puts a cost on somebody. The team has to choose on purpose.
Where should the team set the threshold?
The team picks human review and writes the cost into their design doc: someone has to do this work, carefully.
Who made this?
“In reality, every technology reflects decisions made by designers, companies, and communities, and carries the values and power of those choices.”
- Who benefits from this technology?
- Whose needs are ignored?
A case · you decide
Prei and Reccia read the same filter
Prei & Reccia — a 16-year-old testing a message filter for a coding-club project
On a synthetic test set of 10,000 messages, 200 are genuinely harmful. Sam's filter flags 100 messages, and 80 of those are harmful.
Prei asks: of everything flagged, how much was really harmful? 80 of 100, so precision is 0.8.
Reccia asks: of all the real harm, how much did we catch? 80 of 200, so recall is 0.4.
Sam's slide says his filter is "80% accurate at stopping harm." What is the honest summary?
One number cannot describe a filter. Ask both of Prei's and Reccia's questions.
Who made this?
“In reality, every technology reflects decisions made by designers, companies, and communities, and carries the values and power of those choices.”
- Who benefits from this technology?
- Whose needs are ignored?