ModelRefs / PPO (Proximal Policy Optimization) — AI Glossary

PPO (Proximal Policy Optimization) — AI Glossary

The reinforcement learning algorithm used in RLHF to update the LLM policy against a reward model signal while preventing large policy deviations.

Overview

PPO (Schulman et al. 2017) clips the policy gradient update ratio, preventing destabilizing large steps. In RLHF, the LLM is the policy, the reward model provides the signal, and a KL divergence penalty against the SFT model prevents reward hacking. Used in InstructGPT, ChatGPT, and most early RLHF-trained models. Being replaced by GRPO and DPO for LLMs.

Reference details

Topic	training
Last reviewed	2026-06-24

How it works

The policy generates responses, each is scored, and the update raises the probability of the tokens in better-scoring responses — so the training data is produced by the model being trained.
That makes the method on-policy: once the weights move, previously generated samples no longer reflect the current policy and fresh ones must be drawn.
The update is computed as a ratio between the new and old policy probabilities for each sampled token, multiplied by that token's advantage.
Clipping bounds that ratio to a narrow band around one, and a ratio outside the band contributes no gradient at all — so a single step cannot move any token's probability far, however large its advantage.
Advantages come from the value model's estimate of expected return, which is why a value model exists and why removing it requires a different way to establish a baseline.
A divergence penalty against the frozen reference policy is added to the reward itself rather than to the loss, so the optimiser treats drifting away from the starting model as directly costly.

Example: Four models in memory is why the field moved on

A PPO run holds the policy being trained, a frozen reference copy for the KL penalty, the reward model, and a value model — four sets of weights resident at once, plus their optimizer states. Direct preference optimisation reformulates the objective so the reward and value models disappear, leaving two. That is the practical reason DPO displaced PPO for most alignment work: not that PPO stopped working, but that halving the memory and deleting the reward-model training stage removes most of the operational difficulty. The KL term is what the whole apparatus exists to enforce — a leash to the reference model, so the policy cannot drift somewhere the reward model scores highly and humans do not.

Commonly confused with

PPO is a general reinforcement-learning algorithm; RLHF is the pipeline that uses it with a learned reward model. Confusing them makes it sound as though abandoning PPO means abandoning preference alignment — DPO and GRPO are different optimisers for the same goal. GRPO in particular drops the value model while keeping the sampling-based approach, which is why reasoning training uses it.

When to use it

Reach for it when:

Online preference optimisation, where fresh samples are scored during training
Where a reward model already exists and generating new samples is affordable
Verifiable-reward settings, where the reward is a checker rather than a learned model

Reach for something else when:

Straightforward preference alignment from a fixed dataset — DPO is far simpler
Without a KL penalty tuned and monitored: unleashed optimisation finds reward-model errors
Limited memory, where four resident models is the binding constraint

Primary source

Schulman et al., Proximal Policy Optimization Algorithms (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to PPO (Proximal Policy Optimization) — AI Glossary.

Frequently asked questions

What is PPO (Proximal Policy Optimization)?