What Is RLHF? How Human Feedback Shapes AI Models

New to the underlying method? Start with what reinforcement learning is, which covers the agent, reward and policy vocabulary this article assumes.

What is RLHF?

A pretrained language model learns to predict the next piece of text, which is not the same thing as following an instruction well. The problem is that "following an instruction well" is very hard to write down as a formula.

RLHF sidesteps the formula problem by asking people instead. Show humans several answers, have them say which is better, and learn from those judgments.

The idea predates chatbots. Christiano et al. showed that a reinforcement learning agent could learn from human preferences between pairs of trajectory segments, with no hand-coded reward function. They reported providing feedback on less than one percent of the agent's interactions with the environment, tested on Atari games and simulated robot locomotion.

The same idea was later applied to language models, most visibly in InstructGPT. There the reward is not a game score but a learned stand-in for what people prefer. The short definition lives in the RLHF glossary entry.

The three steps

The clearest public description is the InstructGPT paper by Ouyang et al., whose pipeline has three steps.

1. Supervised fine-tuning. Labelers write demonstrations of the desired behaviour, and the pretrained model is fine-tuned on them. The paper reports that this dataset contains about 13,000 training prompts.
2. Train a reward model. For each prompt the model produces several answers, and labelers rank them. The paper states that each prompt had between four and nine labeled completions, and that it trained on all pairwise comparisons from each prompt as a single batch element. The reward model learns to score answers so that preferred ones score higher.
3. Optimise with reinforcement learning. The language model is tuned with PPO to produce answers the reward model scores highly. The reported dataset is about 31,000 training prompts, taken only from the API.

Two details in step two are worth stating precisely. The loss is the negative log of the sigmoid of the reward difference between the preferred and the rejected completion, averaged over every pair. The paper explains the intuition directly: the difference in rewards represents the log odds that one response will be preferred to the other by a human labeler.

The reward-model dataset has about 33,000 training prompts. The paper also reports using only 6-billion-parameter reward models, because it found that 175-billion-parameter reward model training could be unstable and was therefore less suitable as the value function during reinforcement learning.

Step three carries its own safeguard. A per-token KL penalty from the supervised model is added at each token, which the authors say mitigates over-optimisation of the reward model. A variant called PPO-ptx also mixes pretraining gradients into the PPO gradients, to reduce regressions on public benchmarks.

On the people behind the data: the authors hired a team of about 40 contractors through Upwork and ScaleAI, and report inter-labeler agreement of about 73%. The headline result is that outputs from the 1.3-billion-parameter InstructGPT model were preferred to those of the 175-billion-parameter GPT-3, in the paper's words "despite having 100x fewer parameters".

A runnable reward-model demo

The reward model is the part of RLHF that is easiest to misunderstand, so here is a toy version you can run. It uses no libraries.

Each candidate answer has two features. detail is how much useful, correct content it carries, and flattery is how much it agrees with and flatters the user.

Simulated labelers prefer more detail, but they also like flattery, and the reward model is never told that. It only sees which answer won each comparison, then learns a score using the same pairwise loss form described above.

"""A toy reward model trained from pairwise human preferences (no libraries).

Each candidate answer is described by two features:
  detail    - how much useful, correct detail it has (0 to 1)
  flattery  - how much it agrees with / flatters the user (0 to 1)

Simulated labelers mostly prefer more detail, but they also like flattery.
The reward model only ever sees which answer won, never why.
"""
import math
import random

random.seed(0)

def labeler_prefers_a(a, b):
    """Simulated human: score = detail + 0.6 * flattery, plus a little noise."""
    score = lambda x: x["detail"] + 0.6 * x["flattery"] + random.gauss(0, 0.05)
    return score(a) > score(b)

def random_answer():
    return {"detail": random.random(), "flattery": random.random()}

# 1. Collect 2,000 pairwise comparisons: (winner, loser).
pairs = []
for _ in range(2000):
    a, b = random_answer(), random_answer()
    pairs.append((a, b) if labeler_prefers_a(a, b) else (b, a))

# 2. Train a linear reward model r(x) = w_d * detail + w_f * flattery.
#    Loss per pair = -log(sigmoid(r(winner) - r(loser))), the same form
#    InstructGPT uses for its reward model.
w = {"detail": 0.0, "flattery": 0.0}
r = lambda x: w["detail"] * x["detail"] + w["flattery"] * x["flattery"]
sigmoid = lambda z: 1 / (1 + math.exp(-z))
lr = 0.5

for epoch in range(200):
    for win, lose in pairs:
        p = sigmoid(r(win) - r(lose))
        for k in w:
            w[k] += lr * (1 - p) * (win[k] - lose[k]) / len(pairs)

print("Learned reward weights:", {k: round(v, 2) for k, v in w.items()})

# 3. Use the reward model to pick the best of four candidate answers.
candidates = {
    "careful and accurate": {"detail": 0.9, "flattery": 0.1},
    "accurate and friendly": {"detail": 0.7, "flattery": 0.5},
    "vague but flattering":  {"detail": 0.3, "flattery": 1.0},
    "terse":                 {"detail": 0.4, "flattery": 0.0},
}
ranked = sorted(candidates.items(), key=lambda kv: r(kv[1]), reverse=True)
for name, feats in ranked:
    print(f"  reward {r(feats):6.2f}  {name}")

The output with random.seed(0) on Python 3.13:

Learned reward weights: {'detail': 5.09, 'flattery': 3.09}
  reward   5.10  accurate and friendly
  reward   4.89  careful and accurate
  reward   4.61  vague but flattering
  reward   2.03  terse

Three things are worth noticing in that output.

• The model recovers the hidden bias. The labelers weighted flattery at 0.6 of detail, and the learned weights land at a ratio of about 0.6, which is 3.09 against 5.09. Nobody told the reward model, which inferred the preference from rankings alone.
• It ranks the friendly answer above the most careful one. "Accurate and friendly" scores 5.10 while "careful and accurate" scores 4.89, so the more agreeable answer wins.
• The flatterer is close behind. The vague but flattering answer scores 4.61, within half a point of the top, and an optimiser pushing on whatever the reward model rewards has an obvious direction to push.

We ran this with five other seeds. The learned detail weight stayed between 5.03 and 5.20, the flattery weight between 2.89 and 3.10, and "accurate and friendly" ranked first every time.

This is a teaching toy, so read it as one. Real reward models are language models reading text, not two-number feature vectors, and the 0.6 bias is an assumption chosen for the demo rather than a measurement of any real labeler pool.

What it shows accurately is the mechanism. A reward model learns whatever regularities the rankings contain, including the ones you did not intend to teach it.

RLHF and its alternatives

RLHF is one of several ways to turn preference or feedback signals into model behaviour.

Method	Where the signal comes from	What it changes
RLHF	Human rankings, used to train a reward model	Optimises the model against the reward model with RL, using PPO in InstructGPT
RLAIF, or Constitutional AI	An AI model judging outputs against written principles	Trains a harmless assistant without human labels identifying harmful outputs, in a supervised phase then an RL phase
DPO	The same kind of preference pairs as RLHF	Solves the problem with a simple classification loss, removing the separate reward model, the RL loop and sampling during fine-tuning
RL on verifiable tasks	An automatic check, such as a correct answer or a passing test	The reward is objective, so no human ranking or learned reward model is needed

Each row has a source. The Constitutional AI paper describes training a harmless assistant through self-improvement without any human labels identifying harmful outputs, with the only human oversight provided through a list of principles.

The DPO paper reports solving the RLHF problem with a simple classification loss and eliminating the need for sampling from the model during fine-tuning. It claims performance as good as or better than existing methods, exceeds PPO-based RLHF at controlling sentiment, and matches or improves response quality on summarisation and single-turn dialogue.

The DeepSeek-R1 abstract reports that reasoning can be incentivised through pure reinforcement learning, obviating the need for human-labelled reasoning trajectories, on verifiable tasks such as mathematics, coding competitions and STEM fields.

Which of these a given model uses is usually not fully disclosed. Treat claims about how a specific commercial model was trained with care, unless the vendor has published the details.

Where RLHF goes wrong

• The reward model is a proxy. It predicts human judgments and is not truth itself, so any gap between the two is something an optimiser can exploit. InstructGPT adds its KL penalty specifically to mitigate over-optimisation of the reward model, and the general problem of agents gaming an objective is covered in what reinforcement learning is.
• Sycophancy. Sharma et al. report that five state-of-the-art AI assistants consistently exhibit sycophancy across four varied free-form text-generation tasks. They also find that when a response matches a user's views it is more likely to be preferred, and that optimising outputs against preference models sometimes sacrifices truthfulness in favour of sycophancy.
• Whose preferences? The InstructGPT authors are explicit that they are aligning to their training labelers, to their own preferences as the researchers designing the study, and implicitly to what API customers find valuable. They state that OpenAI's customers are not representative of all potential or current users of language models, and that the labeler group is clearly not representative of the full spectrum of people who will use and be affected by the models.
• It does not make a model safe or correct. The same paper states that its models are "neither fully aligned nor fully safe", and that they still generate toxic or biased outputs and make up facts.

What this means if you build with LLMs

You will almost never run RLHF yourself. You consume models that went through it, and that has practical consequences.

• Expect agreeableness. If your product depends on a model pushing back, as code review, fact-checking and risk assessment all do, test for it. Give the model a wrong premise and see whether it corrects you or goes along with it.
• Do not read confidence as accuracy. A preference-trained model is optimised to produce answers people like, which is not identical to answers that are right.
• Measure behaviour rather than assuming it. Build a small evaluation set that includes pushback cases, and run it whenever you change models. Our tutorials on evaluating prompts programmatically and using an LLM as a judge show how.
• Separate the reward model from the judge model. A reward model is trained once on human rankings to steer training, while a judge model scores outputs at evaluation time. They share techniques but do different jobs.

Sources

Ouyang et al., Training language models to follow instructions with human feedback. The three-step pipeline, the 13k, 33k and 31k dataset sizes, K between 4 and 9, and the reward-model loss. Also the 6B reward model and the 175B instability, about 40 contractors, roughly 73% agreement, the KL penalty and PPO-ptx. Also the 1.3B versus 175B result and the stated limitations, accessed 19 Oct 2026.

Christiano et al., Deep reinforcement learning from human preferences. Learning from pairwise preferences between trajectory segments, with feedback on less than one percent of interactions, accessed 19 Oct 2026.

Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The classification-loss formulation and its reported results, accessed 19 Oct 2026.

Bai et al., Constitutional AI: Harmlessness from AI Feedback. RL from AI feedback, the supervised and RL phases, and oversight through a list of principles, accessed 19 Oct 2026.

Sharma et al., Towards Understanding Sycophancy in Language Models. Sycophancy across five assistants and four tasks, and the role of preference data, accessed 19 Oct 2026.

DeepSeek-AI, DeepSeek-R1. The abstract-level claim on RL for reasoning over verifiable tasks, accessed 19 Oct 2026.

Methodology: the InstructGPT figures were read from the full paper text, not the abstract, which does not carry them. Every other figure was read from the cited paper's own page. The reward-model demo is a ModelRefs teaching toy, executed on Python 3.13.15 across six seeds before publication, and its 0.6 labeler bias is an assumption set in the code rather than a measurement. Nothing here describes how any specific commercial model was trained beyond what the cited papers state.