Reward Model — AI Glossary

Reference details

Related terms

• RLHF (Reinforcement Learning from Human Feedback)
• Direct Preference Optimization (DPO)
• AI Alignment

How it works

1. The scalar is read from the representation at the final token of the response, so the score is a judgement on the completed text rather than an average over its positions.
2. Training minimises the negative log-sigmoid of the difference between the chosen and rejected scores, and because only that difference enters the loss, adding any constant to every score leaves it unchanged.
3. That invariance is the mechanical reason the output has no fixed scale, and it holds per prompt — nothing in training ever relates a score on one prompt to a score on another.
4. Inside a reinforcement-learning loop the scores are therefore whitened per batch, subtracting the batch mean and dividing by its standard deviation, so the optimiser receives a relative signal rather than a drifting absolute one.
5. Several reward models can be trained from different seeds and their disagreement used as an uncertainty estimate, which flags responses far enough from the preference data that none of them should be trusted.
6. That disagreement signal is the available defence against the policy optimising into a region the models score highly and no human would, since the models cannot detect it individually.

Example: It learns differences, not a scale

Training data is pairs: a human saw two responses to one prompt and said A is better than B. The model learns to score A above B — it never sees an absolute rating, so only the *difference* between two scores for the same prompt carries meaning. A score of 2.4 on one prompt and 1.1 on another says nothing about which response was better; they are different scales. This is why reward models are used to rank candidates within a prompt and why comparing raw reward values across prompts, or reading one as a quality percentage, is a category error.

Commonly confused with

A reward model predicts what a human would prefer; it does not know what is correct. Those diverge in a specific direction — preference data rewards confident, fluent, well-formatted answers, so the reward model learns to like them whether or not they are right. An LLM-as-judge does a similar job at evaluation time; the reward model is the version wired into training, where the policy can optimise against its blind spots.

When to use it

Reach for it when:

• Preference-based alignment, as the signal a policy optimises against
• Best-of-n sampling at inference: generate several candidates, keep the top-ranked
• Filtering synthetic training data before fine-tuning on it

Reach for something else when:

• As a quality score to report — the numbers are not calibrated or comparable across prompts
• Optimising hard without a leash: the policy will find the model's errors, not human preference
• Where correctness is verifiable — a test suite or checker is a better reward than a learned proxy

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Reward Model — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path