GRPO (Group Relative Policy Optimization) — AI Glossary

Reference details

Related terms

• PPO (Proximal Policy Optimization)
• RLHF (Reinforcement Learning from Human Feedback)
• Verifiable Reward

How it works

1. For each prompt the policy samples a group of completions, typically eight to sixty-four, rather than the single rollout that policy-gradient methods usually take.
2. Every completion in the group is scored, by a reward model or by a verifiable check such as running tests or comparing against a known answer.
3. The group's own reward mean and standard deviation become the baseline, so each completion's advantage is its reward expressed in standard deviations from its siblings.
4. This is what removes the critic: the baseline that a value network would have estimated is read directly off the group, so no second trainable model exists.
5. Updates then proceed as in PPO, with ratio clipping against the old policy and a KL penalty toward a frozen reference model to limit drift.
6. A structural consequence follows from the baseline being the group: a prompt whose completions all score identically has zero advantage throughout and contributes no gradient at all.

Example: What dropping the critic saves, and what the group baseline costs

Price the memory for a 7B policy under mixed-precision Adam, where a trainable model carries roughly 16 bytes per parameter across master weights, gradients and two optimiser moments: 7e9 x 16 = 112 GB. PPO trains two such models, the policy and the critic, and holds two frozen ones, the reference and the reward model, at about 14 GB each: 2 x 112 + 2 x 14 = 252 GB. GRPO trains one and holds the same two frozen: 112 + 28 = 140 GB. The saving is 112 GB, which is 112 / 252 ≈ 44.4% of the footprint, so GRPO fits the job on 140 GB against PPO's 252 — a factor of 1.8. The group baseline is not free, and its cost shows up as wasted samples. Take a group of eight with binary rewards where four completions succeed: the mean is 0.5 and the population standard deviation is 0.5, so each success carries an advantage of (1 - 0.5) / 0.5 = +1.0 and each failure -1.0, which is an informative update. Now take a prompt where all eight succeed. The mean is 1.0, the standard deviation is 0, every advantage is zero, and the eight completions produced no gradient whatsoever. The same holds when all eight fail. So GRPO learns only from prompts the policy finds genuinely uncertain, and a curriculum of problems that are all too easy or all too hard spends its entire sampling budget on nothing.

Commonly confused with

GRPO is not PPO with a cheaper critic — it has no critic. PPO's value network estimates the expected return from a state, which is a learned function usable on any single rollout; GRPO replaces that estimate with the empirical mean of a group sampled from the same prompt, which requires several completions per prompt and gives an estimate only at the prompt level. That is the real trade: less memory and no critic to go stale, paid for in samples per prompt. It is also distinct from best-of-N or rejection sampling, which also generate a group but keep the best completion as a training target and discard the rest. GRPO trains on the whole group, using the poor completions as the baseline that makes the good ones informative.

When to use it

Reach for it when:

• Where reward is programmatically verifiable — maths answers, test suites, schema validation — which is the setting that made the method prominent.
• When memory is the binding constraint, as removing the critic cut the worked footprint by about 44.4%.
• On problems at the edge of current capability, since prompts with mixed outcomes are the only ones producing gradient.
• When a critic would be hard to train, which is common where reward is sparse and terminal.

Reach for something else when:

• On a curriculum that is uniformly easy or uniformly hard, where every group has zero variance and the sampling budget buys nothing.
• Where sampling is the expensive resource, since it needs a group per prompt rather than one rollout.
• For dense per-step reward, which a value network can exploit and a prompt-level baseline cannot.
• As a drop-in PPO replacement without revisiting the data mix, which now determines whether any learning happens.

Primary source

• Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (arXiv)

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to GRPO (Group Relative Policy Optimization) — AI Glossary.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path