Content Moderation — AI Glossary

Reference details

Related terms

• Guardrails
• Toxicity Detection
• PII Detection

How it works

1. Classifiers run at two points — on the input before it reaches the model, and on the output before it reaches the user — because the two catch different things and neither subsumes the other.
2. Text is scored into harm categories rather than a single verdict, since the action differs by category: some warrant a block, some a warning, some a review queue.
3. Each category carries its own threshold, which is a policy decision setting the false-positive and false-negative rates for that harm independently.
4. The gateway placement is what makes it operationally useful: rules live outside the model, are logged per request, and can be changed in minutes rather than requiring retraining.
5. Low base rates dominate the arithmetic. When violations are a fraction of a percent of traffic, even high precision produces false positives at a rate comparable to true ones, so the threshold must be set against the volume rather than against the classifier's reported accuracy.
6. It is defence in depth rather than a substitute for trained refusal: a filter catches what the model was willing to produce, and trained refusal catches what no filter anticipated.

Example: Why precision matters more than recall at low base rates

Take 1,000,000 requests a month with a 0.1% violation rate: 1,000 genuine violations and 999,000 clean. At 95% recall the filter catches 1,000 x 0.95 = 950 and misses 50. Now the precision side. At 99% precision, the 950 true positives arrive alongside 950 / 0.99 - 950 = 9.6 false positives — about 10 clean requests blocked a month, which nobody will notice. At 90% precision it is 950 / 0.9 - 950 = 105.6, about 106 blocked clean requests. At 70% precision it is 950 / 0.7 - 950 = 407 — four hundred users wrongly blocked to catch the same 950 violations. Recall changed in none of these cases. Recall was identical in all three cases. Precision moved 99 - 70 = 29 points and the wrongly-blocked count moved from 9.6 to 407, a factor of 407 / 9.6 = 42. That is the asymmetry: false positives are drawn from the 999,000 clean requests while true positives come from the 1,000 violations, a pool 999,000 / 1,000 = 999 times larger, so precision is where the user-visible cost of moderation is decided at this base rate. Filtering both input and output doubles the classifier calls to 2,000,000 and doubles both effects.

Commonly confused with

Content moderation is not safety training, and treating them as alternatives produces systems with the weaknesses of both. Training changes the weights, so the behaviour travels with the model, cannot be switched off by a caller, and cannot be updated this afternoon or logged per request. Moderation is a filter outside the model: logged, attributable to a rule, updatable immediately, and bypassed entirely if the model is willing and the filter's categories do not cover the case. The pairing is the point — a filter cannot make a model unwilling, and training cannot respond to an incident today.

When to use it

Reach for it when:

• User-generated input reaching a model, where the filter is the only control that sees the request before the model does.
• Output filtering in regulated or brand-sensitive contexts, where a per-request log of what was blocked and why is required.
• Fast-moving specifics — a named entity, a leaked string, an emergent attack phrasing — which a filter can absorb today.
• As one layer alongside trained refusal, with thresholds set per category against your actual traffic volume.

Reach for something else when:

• As the only control, since a filter's categories are fixed and a willing model will find phrasings outside them.
• With thresholds set from the classifier's reported accuracy rather than your base rate, which is how precision costs get missed.
• Where context determines meaning — quotation, clinical discussion, in-group speech — which a span-level classifier does not see.
• Blocking automatically with no appeal path, given documented classifier bias against dialect and reclaimed language.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Content Moderation — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path