Reinforcement Learning · LLM Reasoning · Alignment
GRPO vs PPO: What You Give Up When You Drop the Critic
GRPO swaps PPO's learned value model for the mean reward of a sampled group. That frees one policy-sized model and made open reasoning RL affordable, at the cost of per-token credit assignment.
The one design change
Both algorithms optimize the same clipped surrogate. A policy generates a response, an advantage says how much better than expected each token was, and the update is clipped so the new policy does not move more than a ratio band of roughly 0.8 to 1.2 from the old one. That band, and the “take the minimum of clipped and unclipped” trick, come straight from the 2017 PPO paper and are unchanged in GRPO.
What differs is where the advantage comes from. PPO trains a value model, usually the same size as the policy, that predicts the expected return from each token position; the advantage is the reward minus that prediction, smoothed with generalized advantage estimation. GRPO, introduced in DeepSeekMath, deletes the value model. For each prompt it samples a group of G answers, scores them, and defines each answer’s advantage as its reward minus the group mean, divided by the group standard deviation. Every token in that answer gets the same advantage. The baseline is no longer learned; it is estimated from siblings.
A second, smaller change follows from the first. PPO for language models, as used in InstructGPT, folds the KL penalty against the reference model into the per-token reward. GRPO adds the KL term directly to the loss, with an estimator that is always positive, because mixing KL into the reward would complicate the group normalization.
Key numbers
| Measurement | PPO | GRPO | Setting and source | Same harness? |
|---|---|---|---|---|
| Models resident during training | Policy, value, reward, reference (4) | Policy, reward, reference (3) | Design, DeepSeekMath Sec. 4.1 | Yes, by construction |
| Advantage granularity | Per token, from the value model | One value per answer, shared by all its tokens | Outcome-supervised GRPO | Yes |
| MATH, 7B, chain of thought | Not run in the paper | 46.8 → 51.7 after RL | DeepSeekMath-Instruct → DeepSeekMath-RL, 144K questions | No PPO arm |
| GSM8K, same run | Not run | 82.9 → 88.2 | DeepSeekMath Table 5 | No PPO arm |
| AIME 2024 pass@1, pure RL from base | Not run | 15.6 → 71.0 | DeepSeek-R1-Zero, rule-based rewards | No PPO arm |
| AIME24/25/26 Avg@16, Qwen3-30B-A3B-Base | Not run | 38.19 (GRPO) vs 49.23 (DPPO) vs 54.79 (CPPO) | CPPO paper, identical data and reward | Yes, GRPO vs later trust-region variants |
| Average Pass@32, R1-Distill-Qwen-1.5B | Not run | 77.41 (GRPO) vs 79.17 (N-GRPO) | N-GRPO paper | Yes, GRPO vs its own variant |
| Reward-vs-KL frontier, sentiment control | Below DPO at equal KL | Not run | DPO paper, PPO as the RLHF baseline | PPO vs DPO only |
| Group size / KL coefficient / policy LR | n/a | 64 / 0.04 / 1e-6 | DeepSeekMath RL setup | n/a |
Read the table for what it does not contain. No paper in this set trains PPO and GRPO under the same data, reward model and compute budget and reports both. DeepSeekMath argues for GRPO from memory and variance, then measures GRPO against its own SFT starting point. Later papers such as CPPO and N-GRPO treat GRPO as the incumbent and beat it, which tells you GRPO is the default, not that it is optimal. The cleanest PPO-versus-alternative number in the literature is the DPO frontier result, and that compares PPO with an offline method, not with GRPO.
What the critic was buying you
PPO’s value model is expensive, but it is not decoration. It gives a baseline per token, so a long answer that goes wrong at step 40 can, in principle, get negative advantage on the tokens after step 40 and positive advantage before it. Outcome-supervised GRPO cannot express that: the whole answer gets one number. DeepSeekMath’s process-supervision variant recovers some of it by scoring each reasoning step with a process reward model and summing normalized step rewards from that step onward, but that reintroduces a trained scorer, which is most of the cost you were trying to avoid.
The value model also absorbs reward noise. When the reward model is a neural network trained on human preferences, as in RLHF for chat, individual scores are noisy and the learned baseline averages that noise across many prompts. GRPO’s baseline is computed from G samples of one prompt. With G = 64 that is a decent estimate; with the G = 8 many open recipes use for cost reasons, the baseline itself is noisy, and normalizing by a small-sample standard deviation amplifies it.
GRPO moves compute rather than removing it. The critic’s forward and backward passes are gone, but you now generate G answers per prompt instead of one, and generation is the slow part of RL for language models. On verifiable tasks that trade is clearly worth it, because sampling is cheap relative to training a second 7B or 70B model, and the reward is exact. On tasks with a learned reward model the accounting is closer.
Why GRPO won anyway
The reason GRPO dominates open reasoning work since early 2025 is the combination it fits: a reward you can compute exactly (an answer key, a unit test, a proof checker), long outputs where the value model would struggle to assign per-token values from a single terminal reward, and memory budgets that cannot hold four large models. DeepSeek-R1 is the existence proof. R1-Zero trains from a base model with GRPO and rule-based rewards only, and AIME 2024 pass@1 climbs from 15.6 to 71.0 with no supervised reasoning data. DeepSeek explicitly avoided a neural reward model to prevent reward hacking, and with an exact reward the critic’s noise-averaging role disappears.
DeepSeekMath’s own analysis explains the mechanism. RL raised Maj@K but not Pass@K: the model did not learn to solve new problems, it learned to put the correct answer at the top of its distribution more often. For that objective a group baseline is enough, because “better than my siblings on this question” is exactly the signal you want.
When to use which
- Verifiable reward, long chain of thought, tight memory: GRPO. This is the R1 regime and the one almost every open reasoning recipe targets.
- Learned preference reward model, chat or writing quality: PPO is still defensible, and the DPO family is the usual alternative. GRPO with a small group and a noisy reward model has fewer safeguards than PPO’s critic gives you.
- Dense or intermediate rewards, agents with multi-step environment feedback: PPO’s per-token baseline maps naturally onto per-step rewards. Outcome GRPO throws that structure away; process GRPO brings back a trained scorer.
- You already have a GRPO codebase and want stability: do not switch to PPO, switch the clip rule. CPPO’s early-token weighting and prefix budget gained 16.6 points over GRPO on the 30B MoE setting under the same data, and N-GRPO’s rollout perturbation gained about 2 points of Pass@32. The trust-region variants are where the recent gains are.
GRPO’s own failure modes
Three problems show up in practice that the original paper does not discuss. First, if every answer in a group gets the same reward, the standard deviation is zero and the prompt contributes no gradient; on easy or impossible prompts that is most of the batch, which is why later recipes filter groups with zero variance or resample until a group is mixed. Second, the per-answer 1/|o| normalization in the GRPO objective weights tokens differently depending on answer length, and analyses in 2025 traced a bias toward longer wrong answers to it. Third, the clip band inherited from PPO caps how much a low-probability token can be pushed up, which starves exploration; the fix in several recipes is to raise the upper clip. CPPO’s diagnosis that early tokens deserve tighter clipping than late tokens is the same family of problem seen from the other side.
Limits and open questions
The biggest gap in this comparison is the missing controlled experiment: a PPO run and a GRPO run with matched data, reward, compute and tuning effort, on both a verifiable task and a learned-reward task. Until that exists, “GRPO matches PPO for less memory” is an inference from design plus the fact that GRPO-trained models are strong, not a measured equivalence. Every GRPO result above uses reasoning benchmarks with exact answers, so nothing here says how GRPO behaves as the main RLHF loop for a chat assistant. And the GRPO numbers come from different base models, group sizes and reward setups; the table marks which rows share a harness so you can avoid comparing 51.7 on MATH with 71.0 on AIME as if they measured the same thing.
FAQ
Is GRPO better than PPO?
Not in a measured, controlled sense. GRPO removes PPO’s value model and estimates the baseline from a group of sampled answers, which cuts memory and worked well enough to train DeepSeek-R1. But no paper in this comparison runs PPO and GRPO under identical conditions, and later trust-region variants beat GRPO by 5 to 16 points on the same data, so GRPO is the practical default rather than the ceiling.
Does GRPO use a critic or value model like PPO?
No. PPO trains a value model, usually as large as the policy, to predict expected return per token. GRPO samples G answers per prompt (64 in DeepSeekMath) and uses each answer’s reward minus the group mean, divided by the group standard deviation, as the advantage for all of its tokens.
How much memory does GRPO save compared with PPO?
PPO keeps four models during training: policy, value, reward and reference. GRPO keeps three, dropping the value model, which is typically the same size as the policy. With rule-based rewards, as in R1-Zero, the reward model disappears too. The saving is one policy-sized model plus its optimizer state, in exchange for generating G samples per prompt.
What is the difference between GRPO and PPO in how the KL penalty is applied?
PPO for language models subtracts a per-token KL penalty inside the reward before advantages are computed. GRPO adds the KL divergence to the reference policy directly in the loss, using an unbiased estimator that is always positive, so group normalization of rewards stays simple.
When should you still use PPO instead of GRPO?
When the reward is a learned preference model with noisy scores, when rewards arrive at intermediate steps rather than only at the end, or when you need per-token credit assignment for very long trajectories. In those cases PPO’s learned baseline does work that a group mean cannot.