Research Papers
EN ZH

Language Models · Open Models · Efficient AI · AI Agents

Nemotron 3 Ultra Explained: Hybrid Mamba-Attention MoE Trained in NVFP4

Nemotron 3 Ultra is a 550B-total, 55B-active hybrid Mamba-attention MoE pretrained in NVFP4 on 20T tokens; it gives up a few points to the strongest open models for up to 5.9x higher decode throughput.

NVIDIA · Paper published 2026-06-12 · 5 min read

Nemotron 3 Ultra Explained: Hybrid Mamba-Attention MoE Trained in NVFP4

Quick answer

Nemotron 3 Ultra is NVIDIA’s June 2026 flagship open model: 550B total parameters, 55B active per token, 108 layers, a hybrid of Mamba-2 state-space layers and attention scaled with LatentMoE experts, and native multi-token prediction with two shared-weight heads. It was pretrained in NVFP4 precision on 20 trillion tokens, extended to a 1M-token context, and post-trained with supervised fine-tuning, multi-environment RL with verifiable rewards, and a two-iteration Multi-teacher On-Policy Distillation (MOPD) stage. The pitch is throughput at parity: on an 8K-input, 64K-output decode-heavy setting it reports 5.9x, 4.8x and 1.6x higher inference throughput than GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B and Qwen-3.5-397B-17B, while its agentic and reasoning scores sit within a few points of those models and behind DeepSeek-V4-Pro on several. Base, post-trained and NVFP4 checkpoints, the training data and the recipe are released.

Architecture at a glance

The layer pattern follows Nemotron 3 Super: most token-mixing layers are Mamba-2 state-space layers, whose per-step decode cost is constant in sequence length, interleaved with a smaller number of attention layers, and the feed-forward side is a LatentMoE. The dimensions in Table 1 explain the throughput: hidden size 8,192; 64 query heads but only 2 key-value heads at head dimension 128, so the attention layers’ KV cache is small; 256 Mamba heads of dimension 64 with state dimension 128 in 8 groups; 512 experts per layer with the top 22 activated and a 2,048-dimensional MoE latent; expert hidden size 5,120 plus a shared expert of intermediate size 10,240. LatentMoE routes tokens to experts through the latent, which NVIDIA reports gives better accuracy per parameter than a standard granular MoE. Figure 14 in the quantization section shows the payoff directly: the Mamba SSM cache is far smaller than an FP8 KV cache would be at the same batch and length.

Multi-token prediction is built in: two MTP heads share parameters, each a single attention layer followed by a single MoE layer, and are used both for speculative decoding at inference and to speed up RL rollouts during post-training, where 3 draft tokens per step cut rollout time relative to no MTP.

NVFP4 pretraining and what went wrong

Pretraining ran entirely in NVFP4 with a warmup-stable-decay schedule, 15T tokens biased toward diversity followed by 5T biased toward quality. Ablations that switch to BF16 from the 5T, 10T and 16T checkpoints show train-loss gaps of 0.33%, 0.34% and 0.03% after 74B further tokens, which is the evidence that the low-precision run tracks a BF16 run. The report is candid about instability: the run diverged twice on the phase-1 data, a rollback to the 15T checkpoint with FP32 reduce-scatter diverged again, starting learning-rate annealing immediately after rollback mitigated it, and the team cut the planned token horizon to 20T. They could not isolate a single cause but document imbalanced and dead experts and imbalanced residual-stream norms as candidates.

Key results

  • Base model (Table 2): Nemotron-3-Ultra-Base against DeepSeek-V3.2-Exp-Base, Mistral-Large-3-675B-Base, Kimi-K2-Base and GLM-4.5-Base: MMLU 89.08 (next best 87.82), MMLU-Pro 79.07 (69.15), AGIEval-En 78.73 (72.55), GPQA 50.00 (43.43), MATH 82.00 (68.40), HumanEval 83.84 (78.20), MBPP-Sanitized 85.97 (84.08); GSM8K 88.10 trails Mistral’s 91.21 and Kimi’s 91.05.
  • Post-training recovery through MOPD (Table 5): Terminal Bench 2.0 goes SFT 34.5 → RLVR 44.5 → MOPD1 50.8 → MOPD2 54.0, past its own 50.0 teacher; SWE-Bench Verified 63.5 → 65.8 → 70.1 → 71.7 (teacher 72.5); TauBench Telecom 55.7 → 82.7 → 91.2 → 92.9 (teacher 94.0); BrowseComp 14.3 → 31.0 → 41.0 → 44.4 (teacher 51.0); GDPVal 23.2 → 28.9 → 46.7 (teacher 49.5).
  • Post-trained model against open models (Table 10): Terminal Bench 2.1 56.4 (GLM-5.1 59.3, Kimi-K2.6 67.2, DeepSeek-V4-Pro 49.2), SWE-Bench Verified 70.7 (GLM-5.1 76.2, Kimi-K2.6 75.7, DS-V4-Pro 74.5), GDPVal 46.7 (GLM-5.1 54.7), BrowseComp 44.4 (Kimi-K2.6 61.3), LiveCodeBench v6 89.0 (DS-V4-Pro 92.5), GPQA 87.0 (Kimi-K2.6 91.0), HLE without tools 26.7 (DS-V4-Pro 37.7), MMLU-Pro 86.8 (Qwen-3.5 88.3), RULER 1M 94.7 (best), IOI 2025 570.0 (Kimi-K2.6 585.0), IFBench 81.7 (DS-V4-Flash 82.0), OmniScience non-hallucination 78.7 (best by a wide margin).
  • Throughput (Figure 1, Figure 15): 5.9x GLM-5.1, 4.8x Kimi-K2.6 and 1.6x Qwen-3.5 at 8K in / 64K out with NVFP4 at max throughput; on a prefill-heavy 50K in / 2K out setting it trails Qwen-3.5, consistent with active parameters governing prefill and total weight I/O governing decode.
  • Quantization: a single NVFP4 checkpoint serves both W4A16 and native W4A4; across a 4.85 to 7.19 bits-per-element sweep only long-context reasoning (AA-LCR) moves, from 62.25 to about 64.7 once mixed-FP8 layers are added.

Limits and open questions

The throughput claims are NVIDIA’s own measurements at maximum throughput on NVIDIA hardware, with speculative decoding enabled where available and the best number chosen per model. On accuracy, Ultra trails the strongest open models on the headline coding and search suites by 5 to 17 points (SWE-Bench Verified 70.7 vs 76.2, BrowseComp 44.4 vs 61.3, HLE 26.7 vs 37.7) and leads mainly on long context, instruction following and non-hallucination. The pretraining instabilities forced a shorter run than planned, and the report does not claim to have found their cause. MOPD’s largest gains are on agentic tasks where the student’s warmup mattered, and the general-reasoning teacher itself trails DeepSeek-V4-Pro on HLE. Where the Mamba-heavy design sits against the delta-attention and compressed-attention answers from Kimi and DeepSeek is in Nemotron 3 Architecture Explained.

FAQ

What is the Nemotron 3 Ultra architecture?

A 108-layer hybrid of Mamba-2 state-space layers and attention layers with 64 query heads and 2 key-value heads, scaled with LatentMoE feed-forward layers of 512 experts with 22 active per token through a 2,048-dimensional latent, for 550B total and 55B active parameters, plus two shared-weight multi-token-prediction heads.

How much faster is Nemotron 3 Ultra than other open models?

NVIDIA reports 5.9x, 4.8x and 1.6x higher decode throughput than GLM-5.1-754B-A40B, Kimi-K2.6-1T-A32B and Qwen-3.5-397B-17B on an 8K-input, 64K-output workload in NVFP4 at maximum throughput. On a prefill-heavy 50K-input workload it trails Qwen-3.5.

How does Nemotron 3 Ultra compare with DeepSeek-V4 and Kimi K2.6 on benchmarks?

It leads on RULER 1M (94.7) and non-hallucination but trails on the hardest agentic and knowledge suites: SWE-Bench Verified 70.7 vs 74.5 (DS-V4-Pro) and 75.7 (Kimi-K2.6), Terminal Bench 2.1 56.4 vs 67.2 (Kimi-K2.6), HLE 26.7 vs 37.7 (DS-V4-Pro), BrowseComp 44.4 vs 61.3 (Kimi-K2.6).

What is multi-teacher on-policy distillation in Nemotron 3?

A post-training stage after SFT and RL where the student’s own outputs are graded against several specialized teachers over two iterations. It lifted Terminal Bench 2.0 from 44.5 to 54.0, past the 50.0 teacher, and SWE-Bench Verified from 65.8 to 71.7 against a 72.5 teacher.