Reinforcement Learning · Language Models · LLM Reasoning
LLM RL Stability vs Forgetting: DRPO and Local Perturbation numbers
DRPO swaps DPPO's hard mask for a smooth Binary-TV weight, stable in 6/6 settings incl. FP8; Local Perturbation Theory shows multi-domain RL forgets via a second-order conflict subspace a short refresh repairs.
Two failure modes that get blamed on the same word
“RL training is unstable” covers two very different breakage patterns, and the fixes do not transfer between them. The first happens inside a single run: the policy drifts from the behavior policy that generated the rollouts, the importance-ratio machinery misbehaves on rare tokens, and training collapses or plateaus. That is the territory of DRPO, which rethinks how the trust region should shape per-token gradients. The second happens across runs: you RL-tune a model on Math, then Code, then QA, then Creative Writing, and the earliest domain quietly rots. That is the territory of Local Perturbation Theory, which explains the forgetting mechanism and tests two cheap repairs.
Both papers are mechanism-first: each opens by showing the standard explanation is wrong, isolates the real cause with measurements, and lands on a fix that costs almost nothing to try. Neither paper runs the other’s experiment, so this page keeps the two number sets in separate columns and never treats them as a head-to-head.
What DRPO fixes: the geometry of a single update
LLM RL is almost always off-policy in practice. Rollouts come from an inference engine whose numerics differ from the training engine, and trajectories are split across mini-batches, so the policy being updated is never quite the one that generated the data. PPO and GRPO control this with clipping on the per-token importance ratio. The argument DRPO inherits from DPPO and Qi et al. is that the ratio is a weak proxy for real distributional shift on a long-tailed vocabulary: on a rare token, a small absolute probability change produces a large ratio change, so a ratio-based trust region overreacts to tail noise and underreacts where the mass is.
The tail is quantified, not hand-waved. On Qwen3-30B-A3B-Base, tokens with behavior probability at or below 0.01 make up 7.8% of all sampled tokens. SPO’s advantage-weighted chi-square penalty scales each squared shift by 1 over the behavior probability, so its per-token gradient weight grows without bound along that low-probability axis. DRPO penalizes the absolute probability shift directly, a squared L2 distance on the Binary-TV value, which is bounded in [0,1] with variance at most one quarter. That boundedness is the whole design.
DPPO had already replaced ratio clipping with a divergence-based mask on the same Binary-TV quantity, but the mask is a hard on/off cutoff: a token’s gradient is zeroed once it crosses the boundary in a harmful direction. DRPO keeps the boundary and smooths the response. The weight decays continuously to zero at the threshold, reverses sign outside it so the gradient pulls the policy back toward the behavior policy, and amplifies updates that move inward. Two ablations carry the evidence. Mask-DRPO, which uses DPPO’s gradient inside the boundary and DRPO’s corrective signal only outside it, matches full DRPO, so most of the gain comes from the correction beyond the trust region rather than from reshaping gradients inside it. Removing the absolute-advantage factor from both SPO and DRPO consistently drops and destabilizes training, because that factor keeps the boundary independent of reward scale. The regularization threshold itself is set to 12.5, with its sensitivity pushed to the appendix.
What Local Perturbation Theory fixes: interference between runs
The standard story for sequential multi-domain RL is catastrophic forgetting caused by gradient conflict. The paper measures the story directly and finds it empty. The global gradient cosine between Math and QA sits near zero, the textbook no-conflict condition, yet training Code, QA, and Creative Writing after Math drags Math from its post-RL peak of 66.49 down to 57.66 on Qwen3-4B-Thinking. Figure 1 resolves the paradox at module level: a most-conflicting module and a most-synergistic module exist at the attention and MLP level, and their effects cancel in the global average. Interference is hidden by aggregation, not absent.
Two measurements anchor the mechanism. First, RL post-training edits are sparse and tiny: for each single-domain expert, 77% to 89% of parameters move by less than 1e-7 in absolute value, with relative changes below 1e-3. Second, the sets of most-changed neurons barely overlap across domains, with Jaccard overlap below 0.19, but the neurons that actually fire during inference overlap heavily across the three reasoning domains. Domains write to different places and read through shared routes, so two near-disjoint updates can still collide on a route they both depend on.
Expanding the earlier domain’s loss to second order around the perturbed parameters, the first-order term is governed by the near-orthogonal gradients and is small. The damage is a second-order term, curvature times update direction, and under the measured sparse-route structure it concentrates in a low-dimensional shared conflict subspace. That concentration is testable: rolling back 2% of MLP neurons selected by a route-conflict criterion recovers 20.4% of the lost Math score, from 59.90 to 61.25, while QA moves only minus 0.06. Extending the rollback to 32% of joint MLP and attention components recovers 73.6% of the loss. The fix the paper recommends for practitioners is even simpler than rollback: re-run a short Math RL pass at the end of the sequence. Math climbs from 57.66 back to 66.04, Code drifts 50.47 to 51.05, QA ticks 62.34 to 62.49, Creative Writing eases 86.52 to 85.96, and the average rises from 64.25 to 66.39. One caveat the authors state themselves: the refresh is re-exposure to data, closer to rehearsal than to parameter-space surgery, so the rollback experiments are the cleaner test of the theory even though they recover less.
Key numbers
| Axis | DRPO (arXiv 2606.09821) | Local Perturbation Theory (arXiv 2606.02398) |
|---|---|---|
| Failure mode | In-run off-policy drift, worst under FP8 | Cross-domain interference across sequential runs |
| Standard story it rejects | Ratio clipping is a valid trust region | Forgetting means global gradient conflict |
| Evidence models | Qwen3-4B-Base, Qwen3-30B-A3B-Base, Qwen3.5-35B-A3B-Base, R1-Distill-Qwen-1.5B | Qwen3-4B-Thinking-2507, judge Qwen3-235B-A22B-Instruct-2507 |
| Headline damage number | Unregularized surrogate drops 0.25 to 0.17 AIME24/25 avg on Qwen3-4B-Base | Math falls 66.49 to 57.66 across Math, Code, QA, CW |
| Where the damage lives | 7.8% of sampled tokens at behavior probability below 0.01 | Second-order term in a shared low-rank conflict subspace |
| Mechanism | Smooth advantage-weighted quadratic on Binary-TV, variance at most 1/4 | Curvature times update direction on co-activated routes |
| Cheap fix | Drop-in per-token gradient weight, threshold 12.5 | Short domain refresh at the end of the sequence |
| Fix recovery | Matches or beats every baseline in 6/6 settings, including FP8-E2E | Math 57.66 to 66.04, average 64.25 to 66.39 |
| Fix side effects | None reported; Mask-DRPO ablation isolates the corrective signal | Code +0.58, QA +0.15, CW -0.56 |
| Data-free repair | None | 2% MLP rollback recovers 20.4% (59.90 to 61.25); 32% joint rollback 73.6% |
| Benchmark scope | Math only, filtered DAPO subset of about 13K problems | Math, Code, QA, Creative Writing, one ordering |
No cell in the right column is comparable to a cell in the left. The models, tasks, metrics, and judges differ, and the page keeps them apart on purpose. What is comparable is the shape of the argument: both papers measure a quantity the field assumed was fine (ratio-based trust regions, global gradient cosine), find it hiding the real damage, and propose a repair that leaves the training pipeline intact.
When to use which
- A single RL run destabilizes or plateaus, especially with FP8 rollouts or MoE models: start with DRPO. GRPO and SPO collapse under FP8 in the paper’s settings, while DRPO trains through them, and the change is a drop-in replacement of the per-token gradient weight, not a new rollout pipeline. If you already train dense models in BF16 and never see instability, expect little upside.
- Sequential domain runs erode an earlier domain, and you cannot merge the data: start with the refresh schedule from Local Perturbation Theory. Reserve a short final pass on the oldest domain; the measured cost on the other domains is under a point each in this study.
- You cannot touch data flow at all: the conflict-subspace rollback is the data-free option, recovering 20.4% of the loss with 2% of MLP neurons and 73.6% with a 32% joint rollback, at the price of needing route-conflict diagnostics the refresh does not.
- You are choosing an RL objective from scratch: neither paper is that guide. Read them after you have picked a recipe such as GRPO or a trust-region variant; for the objective-level comparison, start from the PPO baseline and the trust-region family that flow-DPPO belongs to. The open question of how DRPO behaves on non-verifiable, chat-style rewards remains, as does the interaction between an in-run stabilizer and cross-run interference.
Limits and open questions
DRPO reports AIME24/AIME25 accuracy as training curves, not a per-setting numeric table, so the edge over DPPO is stated as matching or exceeding rather than a quoted gap, and there is no compute or wall-clock accounting. Its benchmarks are math-only with rule-based verification, which says nothing about chat RLHF or agentic tasks. Local Perturbation Theory rests on a single 4B model and one domain ordering, so the claim that the conflict subspace stays low-dimensional at 70B scale, with more domains or a different curriculum, is untested; and its second-order analysis assumes small perturbations, which may fail under aggressive RL with large policy shifts. The most interesting open question sits between the two papers: DRPO stabilizes a run, Local Perturbation Theory heals a sequence, and nobody has measured whether an in-run stabilizer reduces or redistributes cross-run interference.
FAQ
Which paper should I read if my GRPO run collapses under FP8?
DRPO. Its core evidence is that GRPO and SPO collapse in low-precision settings where FP8 plus the MoE architecture widen the train-inference mismatch, while DRPO keeps training and matches or beats every baseline across six settings. The fix is a change to the per-token gradient weight, so it drops into an existing GRPO codebase.
How does the Math refresh in Local Perturbation Theory differ from retraining?
It is a short additional RL pass on Math after the full Math, Code, QA, Creative Writing sequence, not a restart. Math recovers from 57.66 to 66.04 while Code, QA, and Creative Writing each move by less than one point, which reads as selective recovery rather than blunt rehearsal, though the authors note it is still re-exposure to data.
Can the conflict-subspace rollback fix forgetting without any data?
Partially. Rolling back 2% of MLP neurons chosen by a route-conflict criterion recovers 20.4% of the lost Math score and moves QA by only minus 0.06, and a 32% joint MLP plus attention rollback reaches 73.6% recovery. Both are data-free, but neither recovers as much as the refresh, and both need the paper’s diagnostics to select neurons.
What is the difference between DRPO’s trust region and ratio clipping?
Ratio clipping bounds the per-token importance ratio, which explodes on rare tokens where a small absolute shift becomes a large ratio change; DRPO penalizes the absolute Binary-TV shift directly, a quantity bounded in [0,1] with gradient-weight variance at most one quarter. The smooth quadratic also keeps correcting beyond the boundary where a hard mask goes silent, and the Mask-DRPO ablation shows that corrective signal is where most of the gain comes from.
Do these two fixes stack: DRPO inside each run plus a refresh across runs?
Unknown. The papers do not share a harness, model, or task, and neither tests the combination. The plausible interaction is worth an experiment: if DRPO keeps in-run updates tighter to the behavior policy, the second-order damage measured by Local Perturbation Theory could shrink, or could concentrate further. Treat stacking as a hypothesis, not a recommendation.