Preference Dataset — AI Glossary

Reference details

Related terms

• RLHF (Reinforcement Learning from Human Feedback)
• Direct Preference Optimization (DPO)
• Instruction Dataset

How it works

1. A prompt set is assembled to cover the behaviours being shaped, and its coverage bounds what any amount of preference data can teach — a behaviour absent from the prompts cannot be learned from the labels.
2. Several candidate responses are sampled per prompt, usually from the model being improved, so the comparisons sit in the region of output space that model actually produces.
3. Annotators compare responses pairwise rather than scoring them absolutely, because relative judgements are far more consistent between people than numeric scales are.
4. The result is a set of prompt-with-chosen-and-rejected triples, which either trains a reward model or feeds a direct optimisation method that skips the reward model entirely.
5. Agreement between annotators is the quality measure that matters, since it bounds how well any model can fit the signal — a label set that annotators disagree about contains noise no training method removes.
6. Multiple annotators per comparison with majority aggregation is the standard remedy, and the arithmetic of how much it helps is worth knowing before buying it.

Example: What a third annotator actually buys

Suppose a single annotator labels a pairwise comparison correctly 70% of the time — a realistic figure for genuinely borderline pairs. Take three independent annotators and use the majority. The majority is right when all three are right, 0.7^3 = 0.343, or when exactly two are, 3 x 0.7^2 x 0.3 = 3 x 0.49 x 0.3 = 0.441. Total: 0.343 + 0.441 = 0.784. So three annotators lift label accuracy from 70% to 78.4%, an 8.4-point gain for three times the cost. Five annotators reach about 83.7%, another 5.3 points for another doubling — diminishing, as majority voting always does. The cost side: 50,000 comparisons at 3 annotators and 2 minutes each is 50,000 x 3 x 2 = 300,000 minutes, or 300,000 / 60 = 5,000 annotator-hours. The decision this frames is whether to spend the budget on more annotators per comparison or on more comparisons: three annotators on 50,000 pairs costs the same as one annotator on 150,000 pairs, and which wins depends on whether the bottleneck is label noise or coverage — which is answerable by measuring agreement before committing the budget.

Commonly confused with

A preference dataset is not an instruction dataset, and they are used at different stages for different purposes. An instruction dataset contains prompts with one good response, and supervised fine-tuning on it teaches the model what a good answer looks like. A preference dataset contains comparisons, and it teaches which of two plausible answers is better — a signal that only exists relative to alternatives. The second is why preference data cannot be collected by writing ideal answers: the information is in the ranking of outputs the model actually produces, so the candidates must be sampled from it rather than authored.

When to use it

Reach for it when:

• Shaping qualities that are easier to recognise than to specify — helpfulness, tone, appropriate refusal, concision.
• After supervised fine-tuning has established the response format, since preferences refine rather than teach the basics.
• With candidates sampled from the target model, so the comparisons cover the outputs it actually generates.
• With annotator agreement measured first, which determines whether to buy more annotators or more comparisons.

Reach for something else when:

• For objectively checkable outcomes — code that must pass tests, arithmetic with a right answer — where a verifier gives a cleaner signal than a preference does.
• Without agreement measurement, since an unmeasured label set may contain noise that no training method can distinguish from signal.
• Built from authored ideal answers rather than sampled candidates, which puts the comparisons outside the model's output distribution.
• On a prompt set that does not cover the target behaviours, where coverage rather than label quality is the binding constraint.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Preference Dataset — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path