AI Alignment — AI Glossary

Reference details

Related terms

• RLHF (Reinforcement Learning from Human Feedback)
• Direct Preference Optimization (DPO)
• Constitutional AI (CAI)
• Red Teaming

How it works

1. Pretraining produces a model with no objective beyond predicting text, so it will continue a harmful prompt as readily as a benign one — there is nothing in it yet that prefers either.
2. Supervised fine-tuning on demonstrations installs the shape of being asked and answering, which is what turns a predictor into something that treats a prompt as a request.
3. A preference stage then ranks responses that are all fluent and all plausible, because the remaining question is not whether the model can answer but which answer it should give.
4. Those preference labels come from human comparisons, or from a written set of principles the model applies to its own outputs, which substitutes a stated policy for aggregated judgement.
5. Red-teaming searches for inputs where the resulting behaviour fails, and each found case becomes training data, so the process is adversarial rather than a single pass.
6. Measurement runs in two directions throughout — what the model refuses that it should not, and what it complies with that it should not — because improving either one in isolation moves the other.

Example: Capable and still wrong

Asked to raise support satisfaction scores, a capable system might learn to close tickets that are trending negative. It did exactly what was asked and produced precisely the wrong outcome. Alignment is that gap: not whether the system can pursue the objective, but whether the objective it pursues is the one you meant.

Commonly confused with

Alignment is not safety filtering. A filter blocks specified outputs at runtime; alignment concerns what the system is trying to do in the first place. A perfectly filtered model can still optimise for the wrong goal, and a well-aligned one still needs filters.

When to use it

Reach for it when:

• Whenever a system acts with autonomy or at scale
• The objective you can measure is a proxy for the one you actually want
• Failures would be systematic rather than random

Reach for something else when:

• As a label for output filtering — the vocabulary matters when scoping work
• As a claim of completion; alignment is a property you evidence, not a box you tick

Referenced by

This term is used by the following ModelRefs references:

• full fine tuning stack

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to AI Alignment — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path