SDPG: Self-Distilled Policy Gradient for Sparse-Reward RL

Quick answer

SDPG (Self-Distilled Policy Gradient) attacks the sparse-reward problem in RLVR by making the model supervise itself token by token. The same network plays two roles: a student that sees only the question, and a teacher that also sees a privileged hint (the correct answer plus a worked solution from Gemini 2.5 Pro). An auxiliary loss pulls the student’s full-vocabulary distribution toward the teacher’s, giving dense per-token gradient where a verifier reward gives one bit per trajectory. On Qwen3-4B after 400 steps, SDPG-UFKL reaches 0.327 last-checkpoint on AIME2025 against 0.242 for GRPO-style training, 0.380 on AIME2024 against 0.280, and 0.858 on AMC23 against 0.714. The gain is owned by the combination of full-vocabulary on-policy distillation and reference KL regularization, not by the verifier advantage alone.

Why a verifier reward is too thin to learn from

In RLVR, a long chain-of-thought earns a single scalar at the end: right or wrong. That signal has to be smeared back across hundreds of tokens, so most of the trajectory gets almost no information about which token was the good one. SDPG’s premise is that the model already knows more than the reward reveals, you just have to ask it the right way. Condition the same policy on the answer and a reference solution, and its next-token distribution becomes a sharper target than the policy that is guessing blind. The difference between those two distributions is dense supervision that exists at every position, for free in compute terms, since both forward passes run the same weights.

The privileged context c is the crux, and it is also the easy thing to misread. The teacher is not a stronger external model. It is the student conditioned on text it will not have at deployment: The correct answer is {answer}. A common way to solve this is: {solution}. This is the same privileged-teacher trick used in Self-Distilled Agentic RL, applied here to math reasoning rather than agentic tool use.

How the three pieces fit together

The objective stacks three terms. First, a group-relative verifier advantage with a normalized standard deviation, A = (R - mean) / (std + eps_std), where eps_std > 0 stops the denominator from blowing up when every sample in a group scores the same, a known instability in GRPO. Second, the exact full-vocabulary on-policy distillation (OPD) loss: a student-to-teacher KL summed over the entire vocabulary V at each step, D_KL(p_t || stopgrad[q_t]), where p_t is the student distribution and q_t the hint-conditioned teacher. The best-performing variant (UFKL) uses an unnormalized forward KL toward the teacher, pushing the student to cover the teacher’s whole hinted distribution rather than collapse onto its single most likely token. Third, a reference-policy KL term (the URKL variant uses a squared-log penalty with coefficient alpha = 1e-3) that keeps the policy from drifting off the base model.

Two control knobs keep the distillation honest. Positive-advantage gating applies the OPD loss only on trajectories that the verifier marked correct; without it the privileged-but-imperfect teacher can reinforce locally plausible tokens that sit on a globally wrong path. A beta scheduler warms the distillation weight up, holds it, then decays it near the end of training, so the student stops leaning on information it cannot access at test time.

Key results

• AIME2024 (Qwen3-4B, pass@1 mean@32, 400 steps): SDPG-UFKL 0.380 last / 0.408 best vs GRPO 0.280 / 0.316 and RLSD 0.378 / 0.395, so SDPG matches the self-distillation baseline on best and beats GRPO by a full 0.10 on last-checkpoint.
• AIME2025: SDPG-UFKL 0.327 / 0.335 vs GRPO 0.242 / 0.279 and RLSD 0.300 / 0.304, the widest margin: +0.085 last over GRPO and +0.027 over RLSD.
• AMC23: SDPG-UFKL 0.858 / 0.870 vs GRPO 0.714 / 0.739 and RLSD 0.813 / 0.813, a 0.144 last-checkpoint jump over GRPO.
• Entropy stability: SDPG-UFKL holds substantially higher actor entropy through training, while the RLSD self-distillation baseline collapses by step 250, the failure mode SDPG’s KL regularization is built to prevent.
• Speed of separation: in the per-benchmark curves the SDPG variants pull ahead of GRPO and RLSD within the first ~50 steps and reach higher reward plateaus several hundred steps earlier.
• Variant comparison: the unnormalized forward-KL variant (UFKL) edges the reverse-KL variant (URKL) on AIME25 (0.327 vs 0.307 last) and AMC23 best (0.870 vs 0.863), so the KL direction is a real tuning axis, not cosmetic.

What the ablation actually credits

The honest reading is that the verifier advantage is the floor and the distillation is the lift, but only because the KL regularization makes the distillation survivable. RLSD already distills from a privileged teacher and it does beat GRPO, yet it suffers entropy collapse by step 250. SDPG’s contribution is not “add self-distillation,” it is “add full-vocabulary self-distillation and then keep the policy from collapsing into the teacher.” Strip the reference KL and gating and you get RLSD’s instability back. So the gain is a joint effect; reporting the OPD term alone would overstate what one component buys. This sits next to the broader argument in DRPO that the divergence regularizer in LLM RL is doing more load-bearing work than its small coefficient suggests.

Limits and open questions

The privileged context is not free, and the paper says so: it relies on ground-truth answers plus Gemini 2.5 Pro solutions for every training prompt, so SDPG is a fine-tuning method for problems where you already have verified references, not a way to learn from raw verifier signal alone. The authors also flag an irreducible conditional-mutual-information gap: a teacher that conditions on information the deployable student will never see can leave a residual the student cannot close, which is exactly why the beta schedule has to phase the signal out. Evidence is narrow: only Qwen3-4B (plus Qwen3-1.7B in the appendix) and three math contests, with no GSM8K/MATH at scale, no larger models, and no non-math reasoning. There is no wall-clock or token-cost accounting for the second teacher forward pass, and the best-vs-last gap on AIME (0.380 last vs 0.408 best) hints the runs are still noisy at this model size.

FAQ

Does SDPG let a model improve through self-distillation without any external supervision?

No, and this is the most common misread. The “self” in self-distillation means the same weights serve as student and teacher, but the teacher only becomes useful because it conditions on a privileged hint: the ground-truth answer and a worked solution generated by Gemini 2.5 Pro. SDPG is a method for sparse-reward fine-tuning when you already hold verified references for the training set, not free bootstrapping from a verifier reward alone.

Why does SDPG-UFKL avoid the entropy collapse that hits the RLSD baseline at step 250?

RLSD distills from the privileged teacher but has no brake, so the policy keeps concentrating mass and its actor entropy collapses around step 250, a mode-collapse signature. SDPG adds reference-policy KL regularization plus a decaying distillation weight, which keeps the student near the base policy and preserves exploration. In the curves SDPG-UFKL holds substantially higher entropy throughout while still scoring higher, for example 0.327 vs RLSD’s 0.300 on AIME2025 last-checkpoint.

How much does SDPG beat GRPO on the math benchmarks, and on which is the gap widest?

On Qwen3-4B at 400 steps (pass@1, mean@32), SDPG-UFKL beats GRPO by 0.100 on AIME2024 last (0.380 vs 0.280), 0.085 on AIME2025 (0.327 vs 0.242), and 0.144 on AMC23 (0.858 vs 0.714). AMC23 shows the largest absolute jump; AIME2025 shows the widest margin relative to its harder, lower baseline.

What does positive-advantage gating do in the SDPG method?

It applies the on-policy distillation loss only on trajectories the verifier scored correct. Without it, the privileged teacher distribution, which is sharp but still imperfect, can reinforce locally plausible tokens that sit on a globally wrong reasoning path, so distilling on failed trajectories would teach confident mistakes. Gating ties the dense distillation signal to the sparse correctness signal.

One line: SDPG turns a one-bit verifier reward into per-token supervision by letting the same model, conditioned on the answer, teach itself, then uses KL regularization to keep that teaching from collapsing the policy. Read the original paper on arXiv.