Nemotron 3 Architecture Explained: Mamba-Attention Hybrid, LatentMoE, NVFP4

Architecture at a glance

The Nemotron 3 Ultra report describes a 108-layer model with 550B total and 55B active parameters. Token mixing is a hybrid: most layers are Mamba-2 state-space layers, which carry a fixed-size recurrent state whose per-step decode cost does not grow with context, and a minority are attention layers. The attention layers use 64 query heads but only 2 key-value heads at head dimension 128, so even where a KV cache exists it is small. Feed-forward layers are LatentMoE: 512 experts per layer, 22 active per token, routed through a 2,048-dimensional latent, plus a shared expert. Two multi-token-prediction heads with shared weights, each a single attention layer followed by a single MoE layer, draft tokens for speculative decoding and accelerate RL rollouts. The whole thing was pretrained in NVFP4 on 20T tokens with a warmup-stable-decay schedule, then extended to a 1M-token context.

Key numbers

Measurement	Value	Setting and source	Same harness?
Total / active parameters, layers	550B / 55B, 108	Report Table 1	n/a
Attention heads	64 query, 2 key-value, head dim 128	Table 1	n/a
Mamba configuration	256 heads of dim 64, state dim 128, 8 groups	Table 1	n/a
MoE	512 experts, top-22, latent 2,048, expert hidden 5,120, shared expert 10,240	Table 1	n/a
MTP	2 heads, shared weights, attention + MoE each	Section 2.1	n/a
Pretraining	NVFP4, 20T tokens (15T diversity phase + 5T quality phase)	Section 2	n/a
NVFP4 vs BF16 loss gap after 74B extra tokens	0.33%, 0.34%, 0.03% from the 5T, 10T, 16T checkpoints	Figure 3	Yes
Decode throughput, 8K in / 64K out	5.9x GLM-5.1-754B-A40B, 4.8x Kimi-K2.6-1T-A32B, 1.6x Qwen-3.5-397B-17B	Figure 1, NVFP4, max throughput	Yes, NVIDIA measurement
Prefill-heavy, 50K in / 2K out	trails Qwen-3.5	Figure 15	Yes
Base-model MMLU-Pro / MATH / GPQA	79.07 / 82.00 / 50.00 vs next-best open base 69.15 / 68.40 / 43.43	Table 2	Yes
Post-trained SWE-Bench Verified / Terminal Bench 2.1	70.7 / 56.4 vs GLM-5.1 76.2 / 59.3 and Kimi-K2.6 75.7 / 67.2	Table 10	Yes
RULER 1M	94.7, best in the table	Table 10	Yes
MOPD gains	Terminal Bench 2.0 44.5 → 54.0, SWE-Bench Verified 65.8 → 71.7, BrowseComp 31.0 → 44.4	Table 5	Yes

Why a Mamba-heavy hybrid decodes faster

Decode throughput on long outputs is bounded by memory traffic, not arithmetic: every generated token re-reads the KV cache of every attention layer plus the weights. A Mamba layer replaces the growing cache with a fixed-size state, so its per-token cost is flat regardless of how long the conversation or agent trajectory has become. Nemotron 3 makes most layers Mamba-2, keeps a few attention layers for exact retrieval, and shrinks those layers’ cache further with 2 KV heads. The report’s Figure 14 shows the SSM cache staying far below what an FP8 KV cache would occupy at the same batch and length, and the throughput chart follows: 5.9x over GLM-5.1 on an 8K-input, 64K-output workload, the shape of a long agent run.

The same design explains where it loses. Prefill cost scales with active parameters and total compute over the prompt, not with cache reads, so on a 50K-input, 2K-output workload Nemotron 3 Ultra trails Qwen-3.5, which activates fewer parameters. The report says this plainly: active parameters govern prefill, total weight I/O governs decode, and Nemotron 3 optimized the second.

LatentMoE and the MTP heads

LatentMoE routes each token to its 22 experts through a 2,048-dimensional latent rather than the full 8,192 hidden width, which is what makes 512 experts per layer affordable in communication and expert-weight traffic. NVIDIA reports better accuracy per parameter than a standard granular MoE; Kimi K3 adopts the same family with 896 experts and a 3,584-dimensional latent, and adds normalization and a quantile-based router to stabilize it. Nemotron 3 also documents the failure side: during the NVFP4 run the team watched expert load balance and residual-stream norms as health signals and found imbalanced and dead experts around the divergences that forced a rollback and a shortened 20T-token horizon.

The two MTP heads are architecturally light, a single attention layer and a single MoE layer each, and share parameters so the draft is autoregressive across heads. At inference they are the speculative-decoding drafter; at post-training they speed up RL rollout generation, the same lever the Bebop paper studies for keeping MTP acceptance high while policy entropy rises. Nemotron 3’s MTP-boosting stage retrains the heads to close the train-inference mismatch and reports higher accepted lengths on SPEED-Bench.

NVFP4 end to end

Pretraining in 4-bit floating point is the report’s other headline. The evidence for parity is the BF16 switch ablation: continuing 74B tokens in BF16 from the 5T and 10T checkpoints leaves a 0.33% to 0.34% loss gap that does not close, and from the 16T checkpoint the gap is 0.03%. A single NVFP4 checkpoint then serves both as W4A16 and as native W4A4 on Blackwell, and a bits-per-element sweep from 4.85 to 7.19 moves only long-context reasoning, once mixed-FP8 layers are added. The instabilities were real, two divergences on phase-1 data and a failed rollback, but the report does not attribute them to precision.

How it compares with DeepSeek V4 and Kimi K3

Three open flagships took three routes to cheap 1M-token inference. DeepSeek V4 keeps attention in every layer but compresses tokens into fewer KV entries and attends sparsely, reaching 10% of V3.2’s cache. Kimi K3 uses linear delta-rule attention for three of every four layers. Nemotron 3 goes furthest toward recurrence, with Mamba-2 in most layers and only 2 KV heads in the rest. The accuracy table tracks that ordering: on the hardest agentic suites Nemotron 3 trails DeepSeek V4-Pro and Kimi K2.6 by 4 to 17 points (SWE-Bench Verified 70.7 vs 74.5, HLE 26.7 vs 37.7, BrowseComp 44.4 vs 61.3), leads on RULER 1M and non-hallucination, and wins decisively on decode throughput.

Limits and open questions

The exact count of attention versus Mamba layers among the 108 is given by the report’s layer-pattern figure rather than as a number in the text. Throughput is NVIDIA’s own measurement on NVIDIA hardware with the best setting chosen per model. The report does not ablate the hybrid ratio, the 2-KV-head choice or LatentMoE against alternatives at this scale, and the pretraining divergences remain unexplained. And the model’s strongest results depend on the MOPD post-training stage as much as on the architecture.

FAQ

What is the Nemotron 3 Ultra architecture?

A 108-layer hybrid of Mamba-2 state-space layers and attention layers with 64 query heads and 2 key-value heads, scaled with LatentMoE feed-forward layers of 512 experts with 22 active through a 2,048-dimensional latent, for 550B total and 55B active parameters, plus two shared-weight multi-token-prediction heads. It was pretrained in NVFP4 on 20T tokens.

Why is Nemotron 3 faster at decoding than DeepSeek V4 or Kimi K2.6?

Most of its layers are Mamba-2, whose per-token decode cost is constant in context length, and its attention layers have only 2 KV heads, so total weight and cache I/O per generated token is small. On 8K-input, 64K-output workloads NVIDIA reports 5.9x GLM-5.1 and 4.8x Kimi-K2.6; on prefill-heavy workloads it trails Qwen-3.5.

How does Nemotron 3’s architecture compare with Kimi K3’s?

Both use a hybrid of recurrent and attention layers with a latent MoE. Nemotron 3 uses Mamba-2 layers with attention in a minority of layers and 512 experts with 22 active; Kimi K3 uses Kimi Delta Attention in three of every four layers with 896 experts and 16 active, plus Attention Residuals across depth.

Does NVFP4 pretraining hurt Nemotron 3’s accuracy?

The report’s ablation shows a 0.33% to 0.34% training-loss gap versus BF16 from mid-run checkpoints and 0.03% from the 16T checkpoint, and the post-trained model sits within a few points of other open flagships. The run did diverge twice, but the report does not attribute that to precision.