Research Papers
EN ZH

Reinforcement Learning · LLM Reasoning · Language Models

N-GRPO: Semantic Neighbor Mixing for RL Rollouts

N-GRPO perturbs rollout embeddings with semantic neighbors, lifting DeepSeek-R1-Distill-Qwen-1.5B average Pass@32 to 79.17 and AIME25 Pass@32 to 50.28.

Zhejiang University, Ant Group · Paper published 2026-06-09 · 4 min read

N-GRPO: Semantic Neighbor Mixing for RL Rollouts

Quick answer

N-GRPO is a rollout exploration tweak for reasoning RL. Instead of relying only on token sampling or adding random Gaussian noise to embeddings, it mixes an anchor token embedding with embeddings of nearby semantic neighbors. The goal is to make rollouts diverse without pushing the representation off the language manifold. On DeepSeek-R1-Distill-Qwen-1.5B, N-GRPO raises average Pass@32 to 79.17, compared with 77.41 for GRPO and 78.05 for STHT. On AIME25, Pass@32 rises to 50.28, compared with 47.31 for GRPO.

Exploration without semantic drift

The paper’s core complaint is that common rollout diversity is either too shallow or too noisy. Token-level sampling often changes wording without changing the solution path. Direct embedding noise can move a token representation toward unrelated vocabulary regions, which may produce diversity but breaks the reasoning trajectory.

Semantic Neighbor Mixing is the compromise. For selected positions, the method finds nearest semantic neighbors in embedding space and mixes their embeddings with the anchor token. The mixing rate is low, set to 0.1 by default, with k equal to 3 neighbors. The policy still trains under GRPO, but rollout inputs are perturbed in a more structured way.

The result is not huge, but it is consistent enough to matter for training recipes. It also transfers to GSPO in the paper’s additional experiment, where N-GSPO improves average Pass@32 from 77.34 to 79.04 and gives a 7.66 point gain on AIME25.

The method is also careful about when to perturb. It does not replace every token representation. A binary mask decides whether Semantic Neighbor Mixing is active at a position, and ordinary temperature sampling still remains in the generation loop. That detail matters because reasoning traces contain both brittle symbols and flexible language. Over-perturbing every token would likely damage arithmetic notation, variable names, and copied problem facts.

Key results

  • For DeepSeek-R1-Distill-Qwen-1.5B, average Pass@32 is 79.17 with N-GRPO, versus 77.41 for GRPO and 78.05 for STHT.
  • On AIME25 with the same 1.5B backbone, Pass@32 is 50.28 for N-GRPO, 47.31 for GRPO, and 46.73 for STHT.
  • For the 7B distilled backbone, average Pass@32 is 84.20 with N-GRPO, versus 81.94 for GRPO and 82.53 for STHT.
  • On GPQA-Diamond, 1.5B N-GRPO reaches Pass@32 92.87, above base 90.79 and GRPO 91.95.
  • On Qwen3-1.7B-Base, N-GRPO improves AIME25 Pass@32 over GRPO by 5.00 points and reaches the highest overall average Pass@32 in that setting.

What builders should take from it

The practical takeaway is not to copy the headline number blindly. N-GRPO is useful when a team can reproduce the paper’s setup and when the measured bottleneck matches its own product or research loop. The paper-specific evidence above tells builders where the gain comes from, what comparator was used, and which parts are still protocol-dependent. A good follow-up is to rerun the same idea on a local task distribution before treating it as a general capability upgrade.

For RL teams, the paper is most useful as an exploration diagnostic. If grouped rollouts mostly differ in phrasing, the reward signal may see many samples without seeing genuinely different reasoning. If embedding noise produces invalid text or jumps to unrelated tokens, diversity is being bought by semantic damage. N-GRPO gives a middle setting: local movement in embedding space, controlled by a mixing rate and neighbor count.

Limits and open questions

N-GRPO is a training-time exploration method, not an inference trick. It adds nearest-neighbor machinery and new hyperparameters, and the largest gains appear on harder math settings where baseline pass rates leave room. The paper does not prove the same perturbation helps open-ended coding agents, factual QA, or instruction following. The safest takeaway is recipe-level: if GRPO rollouts are redundant, semantic local perturbations may be better than raw embedding noise.

The missing evidence that would change the judgment is a broader external replication: more independent harnesses, clearer release artifacts, and stress tests designed by groups that did not build the method. Until then, the paper is best read as a strong directional result with a concrete evaluation surface.

FAQ

What is N-GRPO?

N-GRPO adds Semantic Neighbor Mixing to GRPO rollouts, perturbing token embeddings toward nearby semantic neighbors instead of adding unconstrained random noise.

How much does N-GRPO improve AIME25?

On DeepSeek-R1-Distill-Qwen-1.5B, AIME25 Pass@32 rises from 47.31 with GRPO to 50.28 with N-GRPO.

Does N-GRPO work outside GRPO?

The paper reports an N-GSPO transfer experiment where average Pass@32 improves from 77.34 to 79.04, suggesting the mixing mechanism is not limited to GRPO.

One line: N-GRPO is interesting because it treats rollout diversity as local semantic movement, not random embedding disturbance. Read the original paper on arXiv.