Reinforcement Learning · LLM Reasoning · Fine-Tuning & Adaptation
Self-Distillation in RL: AntiSD vs SDPG vs SDAR
AntiSD rewards divergence from a privileged teacher (up to +11.5 points, 2-10x fewer steps than GRPO); SDPG distills an answer-hinted teacher (0.327 vs 0.242 AIME25); SDAR gates per-token trust (+10.2 WebShop).
The one trick, and the argument about its sign
All three methods in this comparison rest on the same observation: an RL-trained LLM already knows more than its reward reveals, and you can extract that knowledge by conditioning the same weights on information the deployed model will never see. The teacher is not a bigger model. It is the policy itself, shown a privileged context: a partial solution (AntiSD), the ground-truth answer plus a worked reference (SDPG), or skills retrieved from past experience (SDAR). Because student and teacher share weights, the distillation target costs one extra forward pass per step, not a second model.
Where the three papers genuinely disagree is the sign of the loss. Classical self-distillation minimizes student-teacher divergence: pull the policy toward the privileged distribution. AntiSD’s central claim is that this is backwards. Its diagnostic is that ordinary self-distillation sharpens exactly the wrong tokens, inflating confidence on structural boilerplate while flattening the distribution on the deliberation tokens where exploration matters. AntiSD therefore maximizes a pointwise-mutual-information reward on the high-information tokens instead. SDPG and SDAR stay on the classical minimize side, and their main engineering problem is the failure mode AntiSD is reacting to: an unregularized distillation-to-teacher collapses the policy’s entropy. SDPG’s own baseline, the RLSD method, loses actor entropy around step 250; SDAR’s measurements show the teacher is worse than the student on more than half of all tokens in multi-turn trajectories. Same trick, opposite sign, two different answers to “how do you keep the teacher from wrecking the student.”
None of the three papers evaluates the other two, so the tables below are three separate evidence bases, not a head-to-head. For the base algorithm all three build on, see GRPO vs PPO.
Key numbers
| Measurement | AntiSD | SDPG-UFKL | SDAR | Setting and source | Same harness? |
|---|---|---|---|---|---|
| Sign of teacher signal | Maximize divergence (PMI reward) | Minimize forward KL to teacher | Minimize KL, per-token gated | Each paper’s method | n/a |
| Privileged context | Question + partial solution (self-constructed) | Ground-truth answer + Gemini 2.5 Pro solution | Retrieved skills from past experience | Each paper | n/a |
| Headline gain | Up to +11.5 points over GRPO | +0.100 AIME24 / +0.085 AIME25 / +0.144 AMC23 last-checkpoint over GRPO | +4.7 to +10.2 across three agentic benchmarks on 7B | Each paper | No, different tasks and models |
| Sample efficiency | GRPO accuracy in 2-10x fewer steps; Qwen3-8B/HMMT25 peak in ~1/5 the steps, ends ~+15 | Pulls ahead of GRPO within ~50 steps, higher plateau several hundred steps earlier | Not the paper’s focus | Each paper | No |
| Math benchmarks | AIME 24/25/26, HMMT25, MinervaMath; HumanEval+/MBPP+ | AIME24/25, AMC23 | None (agentic only) | Each paper | No |
| Agentic benchmarks | None (math only) | None | ALFWorld, Search-QA, WebShop | Each paper | No |
| Anti-collapse mechanism | Entropy gate: reward off below 0.93x warmup entropy | Reference KL (alpha = 1e-3) + decaying distillation weight + positive-advantage gating | Sigmoid gate on per-token teacher-student gap | Each paper | n/a |
| Best single data point | Qwen3-8B HMMT25: ~+15 points | Qwen3-4B AIME25: 0.327 vs 0.242 | Qwen3-1.7B WebShop: 38.3 to 58.6 | Each paper | No |
AntiSD: reward where the teacher disagrees
AntiSD computes, per token, the pointwise mutual information between the next token and a privileged context (question plus a hint or partial solution). Where the privileged context moves the distribution, the token carries information; where it does not, the token is structural. Ordinary self-distillation minimizes this gap and thereby, per the paper’s diagnosis, spends gradient on line breaks and “Step” scaffolding while flattening the deliberation tokens. AntiSD flips the objective and maximizes the gap through a bounded shaping function, phi(u) = 0.5 * (softplus(u) - log 2), floor-bounded at -0.5 * log 2 so exploratory tokens cannot be punished without limit.
The engineering core is an entropy-triggered gate. Over a 5-step warmup the method calibrates a baseline teacher entropy H_warm; the PMI reward deactivates once teacher entropy drops below 0.93 * H_warm and re-enables when entropy recovers. The aggressive signal is applied only while it is informative. The paper reports that AntiSD reaches the GRPO baseline accuracy in 2 to 10x fewer training steps across five models from 4B to 30B (Qwen3-4B-Instruct-2507, Qwen3-8B, Qwen3-30B-A3B, OLMo-3-7B-Instruct, OLMo-3-7B-Think), ending up to 11.5 points higher; the strongest single result is Qwen3-8B on HMMT 2025, where AntiSD reaches GRPO’s peak in roughly one-fifth the steps and finishes about 15 points higher.
The honest reservation is breadth: everything is math reasoning, the regime where a privileged partial solution is easy to build, and the constants (0.93, the 5-step warmup, the phi bound) read as tuned. For a method that needs no external teacher data (the privileged context is self-constructed), that is a reasonable trade.
SDPG: full-vocabulary distillation that survives its own teacher
SDPG keeps the classical minimize direction and solves the collapse problem with regularization instead of sign-flipping. The teacher sees a templated hint: The correct answer is {answer}. A common way to solve this is: {solution}, where the solution comes from Gemini 2.5 Pro. The loss is an exact full-vocabulary on-policy distillation KL summed over the whole vocabulary at each step (the unnormalized forward-KL variant, UFKL, is the best performer), stacked on a group-relative verifier advantage with normalized standard deviation, plus a reference-policy KL term with coefficient alpha = 1e-3.
Three knobs keep the distillation honest. Positive-advantage gating applies the distillation loss only on trajectories the verifier scored correct, so a sharp-but-imperfect teacher cannot reinforce plausible tokens on globally wrong paths. A beta scheduler warms up, holds, then decays the distillation weight, phasing out information the student cannot access at test time. And the reference KL is what separates SDPG from its own ablation baseline RLSD, which distills from the same privileged teacher but collapses in actor entropy around step 250. On Qwen3-4B at 400 steps (pass@1, mean@32), SDPG-UFKL reaches 0.380 last-checkpoint on AIME 2024 versus 0.280 for GRPO, 0.327 on AIME 2025 versus 0.242, and 0.858 on AMC 23 versus 0.714. The KL direction matters as a tuning axis: reverse-KL (URKL) lands at 0.307 on AIME 2025 last versus 0.327 for UFKL.
The caveat is data dependence. Unlike AntiSD, SDPG needs verified ground-truth answers and external reference solutions for every training prompt, so it is a fine-tuning method for problems where you already hold references, not a way to learn from raw verifier signal alone.
SDAR: distillation is a gated helper, not a co-pilot
SDAR moves the same trick to multi-turn agents, where the failure mode is worse. The teacher is the same policy given skills retrieved from past experience (context the student never sees at inference), and naive distillation on top of GRPO breaks for two reasons: teacher-student drift compounds across turns, and because retrieval is imperfect, more than 50% of tokens actually show a negative teacher-student log-prob gap. Ungated, you are copying a teacher that is worse than the student most of the time.
SDAR’s answer is a sigmoid gate over the per-token gap: amplify distillation where the teacher is genuinely better, attenuate it elsewhere, with RL kept as the primary objective (L = L_GRPO + lambda * L_SDAR, reported sweet spot beta = 5.0, lambda = 0.01). The results, success rate or accuracy of SDAR versus its GRPO baseline: Qwen2.5-3B-Instruct improves on ALFWorld 75.0 to 84.4 (+9.4), Search-QA 36.4 to 43.4 (+7.0), WebShop 63.3 to 68.0 (+4.7); Qwen2.5-7B-Instruct improves on ALFWorld 81.2 to 85.9 (+4.7), Search-QA 42.0 to 49.0 (+7.0), WebShop 72.6 to 82.8 (+10.2); Qwen3-1.7B’s WebShop jumps 38.3 to 58.6 (+20.3), the largest single gain, though from the weakest baseline. The gate, not the skills, does the work: gains survive even random retrieval (+1.9 on ALFWorld).
When to use which
The three methods map onto three distinct diagnoses. If your problem is that RLVR gives one bit per trajectory and you want dense per-token signal on math problems where you already hold verified references, SDPG’s recipe is the most directly validated. If you want the sample-efficiency gain without any external solution data, AntiSD’s sign-flip plus entropy gate is the only one of the three that constructs its own privileged context. If you are training multi-turn agents and have retrievable past experience, SDAR’s per-token gate is the only design here that has measured, and compensated for, a teacher that is wrong on a majority of tokens.
Limits and open questions
The headline figures above come from three different evidence bases: AntiSD from math reasoning on five models from 4B to 30B, SDPG from Qwen3-4B on three math contests with Gemini 2.5 Pro reference solutions, SDAR from Qwen models up to 7B on three agentic benchmarks. No pair shares a harness, so the numbers establish that each method beats its own GRPO baseline, not that one method dominates another. Open questions cut across all three: how sensitive AntiSD’s gate constants are outside math; whether SDPG’s gains survive without external reference solutions; whether SDAR’s internalization-of-skills claim holds under stress tests; and, for all of them, whether the gains survive on models from families other than Qwen and OLMo. What none of the three supports is the ungated version: every paper’s ablations or diagnostics converge on the same finding, that distilling from a privileged same-weights teacher without a gate, a schedule, or a KL brake will eventually sharpen the wrong tokens or collapse the policy’s entropy.
FAQ
What is privileged-teacher self-distillation in RL?
It is a family of methods where the same model serves as both student and teacher. The teacher forward pass is conditioned on extra context the deployed model will not see (a partial solution, the ground-truth answer, or retrieved skills), making its per-token distribution a dense, cheap supervision target for RL on top of a sparse end-of-trajectory reward.
AntiSD maximizes divergence while SDPG and SDAR minimize it: which is right?
They are answering different diagnoses. AntiSD’s measurement says classical self-distillation sharpens structural tokens and flattens deliberation tokens, so it rewards divergence on high-information tokens instead. SDPG and SDAR accept the classical direction and instead solve the collapse and bad-teacher problems with reference KL, schedules, and per-token gating. There is no head-to-head benchmark between them.
How much do AntiSD, SDPG, and SDAR each beat GRPO by?
Per each paper’s own setup: AntiSD reaches GRPO accuracy in 2-10x fewer steps and ends up to 11.5 points higher on math. SDPG-UFKL on Qwen3-4B beats GRPO by +0.100 (AIME24), +0.085 (AIME25), and +0.144 (AMC23) at last-checkpoint. SDAR on Qwen2.5-7B adds +4.7 to +10.2 across ALFWorld, Search-QA, and WebShop. These are different models, tasks, and harnesses, not a single ranking.
Why do AntiSD, SDPG, and SDAR all need a gate or a schedule?
Because an ungated privileged-teacher loss demonstrably hurts. SDPG’s RLSD baseline loses actor entropy around step 250 without a KL brake; SDAR measures a negative teacher-student gap on more than 50% of tokens in multi-turn settings; AntiSD shows classical distillation flattens exactly the exploration tokens that matter. The gate, schedule, or sign-flip is the contribution, not a tuning detail.
One line: give the same model a peek at the answer, and either reward where that peek changes its mind (AntiSD), distill it with a brake on (SDPG), or copy it only where it is actually right (SDAR). Read the AntiSD, SDPG, and SDAR paper pages for the per-paper detail.