DeepSeekMath Explained: The Paper That Introduced GRPO

Quick answer

DeepSeekMath is the February 2024 DeepSeek paper that did two things the field still uses. It showed that a 7B model can reach 51.7% on competition-level MATH without tools or majority voting, close to GPT-4’s 52.9% at the time, mostly by continuing pretraining on 120B tokens of math scraped from Common Crawl. And it introduced Group Relative Policy Optimization (GRPO), a variant of PPO that removes the value model and instead scores each sampled answer against the mean of a group of answers to the same question. GRPO is the algorithm behind DeepSeek-R1 and most open reasoning-RL work since.

Where the 51.7% actually comes from

The paper is often cited for GRPO, but the pretraining corpus does most of the lifting. DeepSeek built an iterative pipeline: seed with OpenWebMath, train a fastText classifier to find math-like pages in Common Crawl, harvest whole domains that score well, hand-label URL patterns, and repeat. Four rounds yielded 35.5M pages and 120B math tokens, roughly seven times Minerva’s math data and nine times OpenWebMath. Continued pretraining of DeepSeek-Coder-Base-v1.5 7B on a mix of this corpus plus code and general text produced DeepSeekMath-Base 7B, which already scored 64.2% on GSM8K and 36.2% on MATH few-shot, ahead of Minerva 540B.

Two negative results from the ablations are worth remembering. Training on arXiv papers alone did not improve any math benchmark the authors tested. Training on code first did help, both with and without tool use, which is one of the cleaner pieces of evidence for the “code training improves reasoning” folk claim.

Supervised fine-tuning on 776K math instruction examples, including chain-of-thought, program-of-thought and tool-integrated formats, then gave DeepSeekMath-Instruct 7B: 82.9% on GSM8K and 46.8% on MATH.

From PPO to GRPO

PPO for language models is an actor-critic loop. A policy generates an answer, a reward model scores it, and a separate value model, usually the same size as the policy, estimates a per-token baseline so that advantages have low variance. That second model is expensive in memory and awkward to train, because in the LLM setting only the final token gets a real reward and the value model has to guess intermediate values.

GRPO removes the value model. For each question it samples a group of outputs (64 in the paper), scores each with the reward model, and normalizes the rewards within the group: subtract the group mean, divide by the group standard deviation. That normalized reward is the advantage for every token of that output. The clipped surrogate objective is otherwise the same as PPO’s. Two details differ. The KL penalty against the reference model is added directly to the loss rather than folded into the per-token reward, and it uses an unbiased estimator that is always positive. And the paper frames the group-relative baseline as matching how reward models are trained, on comparisons between answers to the same question.

The paper also tests a process-supervision variant, where a process reward model scores each reasoning step and the advantage of a token is the sum of normalized step rewards from that step onward, and an iterative variant where the reward model is retrained on fresh policy samples with a replay buffer.

Key results

• RL stage, chain-of-thought, no tools (Table 5): DeepSeekMath-RL 7B scores 88.2% on GSM8K and 51.7% on MATH, up from 82.9% and 46.8% for the Instruct model it started from. Training used only about 144K GSM8K and MATH questions from the SFT set.
• Out-of-domain transfer: CMATH rises from 84.6% to 88.8% and MGSM-zh from 73.2% to 79.6%, even though neither dataset was in the RL data.
• Against larger models: at 51.7% MATH, the 7B model beats every open model from 7B to 70B in the paper’s table, including InternLM2-Math 20B at 37.7% and Qwen 72B at 35.2%, and sits between GPT-4 (52.9%) and Inflection-2 (34.8%).
• With majority voting: self-consistency over 64 samples lifts MATH to 60.9%.
• Tool-integrated reasoning: 58.8% on MATH with Python, up from 57.4% for Instruct.
• Training setup: policy learning rate 1e-6, KL coefficient 0.04, 64 samples per question, max length 1024, batch 1024, one policy update per exploration step.

What RL changed, and what it did not

The most useful analysis in the paper is Figure 7. The authors measure Pass@K and Maj@K for the Instruct and RL models. RL improves Maj@K but not Pass@K. In plain terms, the RL model is not solving problems the Instruct model could never solve; it is putting the correct answer at the top of the distribution more often. The authors say it directly: the gain comes from making the output distribution steadier, not from new fundamental capability. Anyone quoting GRPO gains should keep that framing, because it also explains why RL on verifiable rewards works so well on math and code, where “pick the right answer more consistently” is exactly what the benchmark measures.

Limits and open questions

The RL evidence is narrow: one 7B model, one reward model trained from DeepSeekMath-Base, and training questions drawn from the same two benchmarks that report the headline numbers. Out-of-domain gains on CMATH and MGSM-zh are encouraging but modest. The paper does not run a controlled PPO baseline with the same data and reward model, so “GRPO matches PPO at lower memory” is an argument from design, not a measured result in this paper; later work such as CPPO and N-GRPO treats GRPO as the baseline to beat rather than re-running that comparison. Process supervision and iterative RL are reported as improvements in the text, but their numbers are less prominent than the outcome-supervision headline. And the group-normalized advantage has a known failure mode the paper does not discuss: when every sample in a group gets the same reward, the standard deviation is zero and the question contributes no gradient, which is why later GRPO variants add filtering or dynamic sampling.

Should you read the paper or this page

Read the paper if you are implementing GRPO, because Section 4 and Appendix A give the objective, the advantage formulas and a unified view of SFT, rejection sampling, DPO, PPO and GRPO as one gradient template. If you want the argument for group-relative advantages versus a learned critic, see the head-to-head in GRPO vs PPO. If you only need the numbers, the table above is the paper.

FAQ

What is GRPO in the DeepSeekMath paper?

GRPO, Group Relative Policy Optimization, is a PPO variant that drops the value model. For each question it samples a group of answers, scores them with a reward model, and uses each answer’s reward minus the group mean, divided by the group standard deviation, as the advantage for all of its tokens. The KL penalty to the reference model goes into the loss rather than the reward.

How much did GRPO improve DeepSeekMath’s MATH and GSM8K results?

Starting from DeepSeekMath-Instruct 7B at 46.8% MATH and 82.9% GSM8K, one RL stage with GRPO on about 144K questions produced DeepSeekMath-RL 7B at 51.7% MATH and 88.2% GSM8K, with chain-of-thought reasoning and no tools.

Does DeepSeekMath compare GRPO against PPO directly?

No. The paper motivates GRPO by memory and variance arguments and gives a unified paradigm that includes PPO, but it does not train a PPO baseline under the same data and reward model. The 51.7% number is GRPO versus the SFT starting point, not GRPO versus PPO.

Why does the paper say RL improves Maj@K but not Pass@K?

Because it measured both. After RL, the chance that the majority-voted answer is correct goes up, while the chance that at least one of K samples is correct does not. The authors read this as RL sharpening the output distribution rather than adding new problem-solving ability.