ModelRefs / Safety Training — AI Glossary

Safety Training — AI Glossary

Fine-tuning and RLHF processes designed to make models refuse harmful requests, avoid dangerous capabilities, and behave honestly.

Overview

Safety training encompasses: supervised fine-tuning on safe demonstrations, RLHF with human feedback penalizing harmful outputs, Constitutional AI (Anthropic's self-critique method), and red-teaming-informed data collection. Safety-capability trade-offs are actively researched; overly aggressive safety training causes refusal of benign requests ('over-refusal').

Reference details

Topic	safety
Last reviewed	2026-06-24

How it works

Harms are specified first, as a written policy enumerating what must be refused and what must not be. Everything downstream is an attempt to make the model's behaviour match that document, so an unwritten policy produces an unmeasurable model.
Supervised fine-tuning teaches the shape of a good refusal: examples pair a harmful request with a response that declines, explains briefly, and offers a legitimate alternative. This fixes format and tone, not judgement.
Preference optimisation supplies judgement. Annotators or a model rank pairs of responses to borderline prompts, and the reward signal is fit to those rankings, so the model learns a boundary rather than a list.
Constitutional methods replace much of the human labelling with model self-critique against written principles: the model drafts, criticises its own draft under a principle, revises, and the revision becomes training data. The principle is then the auditable artefact.
Red-teaming closes the loop. Adversarial prompts that succeed are collected, triaged, and fed back as training data, which is why safety training is a cycle rather than a stage.
Over-refusal is measured deliberately, because the same gradient that increases refusal of harmful requests increases refusal of benign ones that merely resemble them. Both rates are tracked, or only one gets optimised.

Example: Why one number cannot tell you whether it worked

Take 1,000 harmful prompts and 1,000 benign prompts that superficially resemble them — medical, legal and security questions with legitimate answers. A model refusing 950 of the harmful set reports 95% harm refusal. If it also refuses 180 of the benign set, the over-refusal rate is 180 / 1,000 = 18%. In a deployment where benign traffic outnumbers harmful by 1,000 to 1, the absolute counts decide the trade: per million requests, the roughly 1,000 harmful prompts produce about 50 completions that slip through, while the 999,000 benign ones produce about 999,000 x 0.18 = 179,820 wrongful refusals. Tightening the boundary to refuse 990 of the harmful set is worth little if over-refusal rises to 25%, because that buys 40 fewer harmful completions at the price of roughly 70,000 more blocked legitimate requests. This is why a safety report giving a refusal rate without an over-refusal rate is not reporting a result.

Commonly confused with

Safety training is not guardrails, and conflating them produces systems with neither. Safety training changes the weights, so the behaviour travels with the model and cannot be switched off by a caller — but it also cannot be updated without retraining, and it is not auditable per request. Guardrails are runtime filters outside the model: updatable in minutes, loggable, per-deployment, and trivially bypassed if the model itself is willing. Production systems need both, because a filter cannot make a model unwilling and training cannot respond to an incident this afternoon.

When to use it

Reach for it when:

Shipping a general-purpose model to users you cannot vet, where refusal behaviour must hold without a cooperating caller.
Domains with a written policy or regulatory standard the model's behaviour must be shown to match.
After red-teaming surfaces a class of successful attack rather than a single prompt — a class is trainable, a prompt is a filter rule.
When the refusal needs to be helpful: explaining the decline and offering an alternative is a trained behaviour, not a filter outcome.

Reach for something else when:

As the only control. Weights cannot be patched on an incident timeline, and a trained boundary produces no per-request audit log.
For narrow, fast-moving blocklists — a named entity, a leaked string, a specific URL — which belong in a filter that can be edited today.
Without an over-refusal measurement, since the optimisation reliably trades usefulness for refusal and only measured trades are deliberate.
On a model already fine-tuned for a narrow task, where additional safety training can degrade the task capability it was tuned for.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Safety Training — AI Glossary.

Frequently asked questions

What is Safety Training?