ModelRefs / RLHF (Reinforcement Learning from Human Feedback) — AI Glossary
RLHF (Reinforcement Learning from Human Feedback) — AI Glossary
An alignment pipeline that trains a reward model from human preference comparisons and optimizes the LLM policy against it. Also called RLHF.
Overview
The classic RLHF pipeline (SFT → reward model → PPO) made ChatGPT possible. Modern stacks replace PPO with DPO or KTO for simplicity and stability. Human preference collection is the most expensive part.
Reference details
| Topic | training |
|---|---|
| Also known as | RLHF |
| Last reviewed | 2026-06-24 |
Related terms
How it works
- The pipeline starts from a supervised fine-tuned model, because preferences can only be collected over responses that are already fluent enough to compare.
- A reward model is initialised from that same checkpoint with its output layer replaced by a single scalar head, so it inherits the language understanding rather than learning it again.
- It is trained on pairwise comparisons with a loss that only requires the chosen response to score above the rejected one, which is why no absolute quality scale is ever needed.
- The policy is then optimised to raise that scalar, sampling its own responses and updating toward whichever score higher — so unlike supervised training, the data is generated by the model being trained.
- A penalty on divergence from the starting policy is added to the objective, and this term is what prevents collapse: without it the policy drifts into degenerate text that scores well and reads badly.
- The strength of that penalty is therefore the central tuning decision, trading how far behaviour can move against how recognisable the result remains as language.
Example: What the human actually provides
Annotators rarely write ideal answers. They see two model responses to the same prompt and pick the better one. Those comparisons train a reward model that scores responses, and the policy is then optimised against that score. The pipeline is built on preferences because ranking two options is far more reliable and cheaper than authoring a gold answer.
Commonly confused with
RLHF optimises a learned proxy for human preference, not human preference itself. Push hard enough and the policy exploits the reward model — the classic result being answers that are longer, more hedged and more agreeable because those correlate with approval in the training data.
When to use it
Reach for it when:
- You need broad behavioural shaping across open-ended tasks
- Quality is a judgement call rather than a checkable answer
- You can fund and sustain preference collection
Reach for something else when:
- Correctness is verifiable — train against the verifier instead
- You lack the annotation budget; a weak reward model is worse than none
- DPO or a similar direct method would achieve the same with far less machinery
Referenced by
This term is used by the following ModelRefs references:
Continue your research
Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to RLHF (Reinforcement Learning from Human Feedback) — AI Glossary.
Frequently asked questions
What is RLHF (Reinforcement Learning from Human Feedback)?
An alignment pipeline that trains a reward model from human preference comparisons and optimizes the LLM policy against it.
Is RLHF (Reinforcement Learning from Human Feedback) the same as RLHF?
Yes — RLHF are common aliases for RLHF (Reinforcement Learning from Human Feedback).
What concepts are related to RLHF (Reinforcement Learning from Human Feedback)?
Closely related concepts include sft, dpo, reward model, alignment.