Language Models · Efficient AI · Open Models · Long Context
Qwen3.8-Next Architecture Explained: GDN, QSA, Gated Residual, N-gram
Qwen3.8-Flash-Next activates 6B of 125B parameters plus 51B host-memory n-gram tables; a 3:1 GDN hybrid, sparse attention, a gated four-branch residual and Muon give Qwen3.7-Plus quality at a ninth of the FLOPs.
Architecture at a glance
The Qwen3.8-Next paper is a design record for Qwen3.8-Flash-Next: 125B total parameters, 6B activated per token, plus 51B parameters of n-gram embedding tables held in host memory and prefetched. Four components carry it. Token mixing alternates three Gated DeltaNet (GDN) layers with one attention layer; at continued pretraining the attention layers become Qwen Sparse Attention (QSA). Every sublayer reads from and writes to a residual stream widened to four branches through an elementwise gate, the Gated Residual. A single n-gram embedding layer, placed at layer 2, adds capacity outside the backbone. Muon trains the two-dimensional weights, with embeddings, the output head and the MoE router on AdamW. The base model leads the 397B-A17B Qwen3.7-Plus on 8 of 14 pretraining benchmarks and trails on the rest by at most 2.6 points, at about a third of the activated parameters and tokens and roughly a ninth of the training FLOPs.
Key numbers
| Measurement | Value | Setting and source | Same harness? |
|---|---|---|---|
| Total / active / n-gram parameters | 125B / 6B / 51B | Paper abstract | n/a |
| Token-mixing ablation, 9-benchmark average | full attention 49.87; SWA hybrid 51.15; GDN hybrid 53.81 | 28-layer 25B-A3B, 400B + 80B tokens, Table 1 | Yes |
| Same ablation, MATH / BBH / MultiPL-E | 49.40 / 63.78 / 39.73 → 53.98 / 68.72 / 47.48 (full → GDN) | Table 1 | Yes |
| QSA vs full attention, short-context average | 75.9 → 76.8 | Production model, Table 2 | Yes |
| QSA vs full attention, RULER 512K to 1M | 90.08 → 93.00 | Table 3 | Yes |
| QSA vs full attention, 8-needle MRCR at 512K / 1M | 30.66 / 20.71 → 40.53 / 26.44 | Table 3 | Yes |
| MTP accepted length with QSA | 4.06 → 4.07 | Four-step drafting, Table 4 | Yes |
| Base model vs Qwen3.7-Plus-Base, MMLU-Pro / SuperGPQA / SWEBench-Pretrain | 73.23 vs 70.90 / 51.36 vs 48.42 / 50.99 vs 49.24 | Table 11 | Yes |
| Where it trails Qwen3.7-Plus | MATH 72.78 vs 74.38; MultiPL-E 79.09 vs 81.68 | Table 11 | Yes |
| Compute ratio vs Qwen3.7-Plus | about 1/3 active params, 1/3 tokens, 1/9 FLOPs | Paper Section 4 | n/a |
| Stability stress test | Muon spikes more than AdamW at constant LR; Muon plus Gated Residual spikes least | 25B-A3B, Figures 10 to 12 | Yes |
Token mixing: the ablation that justifies 3:1
GDN is a gated delta-rule linear attention in the same family as Mamba variants and Kimi Delta Attention: a fixed-size state, updated by writing the difference between the new value and what the state predicts, with a decay gate. The subquadratic-architectures study explains what such gating can and cannot compute, and the Qwen paper supplies the deployment-scale evidence. Three 25B-A3B checkpoints trained on the same tokens, differing only in token mixing, average 49.87 for full attention, 51.15 for a 128-token sliding-window hybrid and 53.81 for the GDN hybrid, with both hybrids keeping one full-attention layer in four. The GDN gains land on MATH, BBH and code, the same places Muon’s gains land in Moonlight, and the paper’s stated reason for keeping the one attention layer is exact token-level retrieval, which no finite-state memory reproduces.
Kimi K3 arrives at the identical 3:1 split with KDA but publishes no ablation; Qwen’s table is the number both designs lean on.
Qwen Sparse Attention
At continued pretraining the full-attention layers are swapped for QSA. A lightweight indexer compresses the sequence into micro-block representations, scores them under a compressed causal mask, and selects the top-k blocks; the selected indices expand into a micro-block sparse mask for the core attention. The design target is that indexer cost itself falls with sequence length, which the paper contrasts with sparse designs whose indexer overhead stays significant at long context. The swap is close to free on short context (75.9 to 76.8 average) and positive at long context: RULER past 512K rises from 90.08 to 93.00 and 8-needle MRCR at 1M from 20.71 to 26.44, while MTP acceptance length is unchanged. Compared with DeepSeek V4, which compresses tokens into single KV entries before sparse selection to cut the cache itself, QSA keeps full-resolution KV and saves indexing and attention compute; Qwen’s KV savings come from the GDN layers instead.
The residual stream
Widening the residual stream to several branches is a 2026 trend with three implementations. DeepSeek V4’s mHC uses learned mixing matrices constrained to be doubly stochastic. Kimi K3’s Attention Residuals attend over earlier layers. Qwen’s Gated Residual widens to four branches and reads them through an elementwise gate with a group RMSNorm, then writes back; its ablation on 25B-A3B models at 560B tokens finds that once read and write are expressive enough, the explicit mixing operator adds little, and that the gate is what supplies the rescaling that keeps training stable. Reads and writes are fused kernels, so the widened stream costs one traversal per block each way. Figure 7 shows the gate learning cross-layer paths, a cheap echo of what AttnRes does explicitly.
N-gram embeddings off the accelerator
The single n-gram embedding layer conditions lookups on local context rather than token identity alone. Its 51B parameters live in host memory and are prefetched, so per-token FLOPs and latency barely move. Two findings temper the idea: placement matters (Table 7), and enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates (Tables 8 and 9), one of the paper’s examples of loss and accuracy disagreeing. Several tokenization and allocation variants gave no consistent gain.
Optimizer and stability
Muon is the main optimizer on attention and GDN projections, expert MLPs and the n-gram key and value projections. Embeddings, the output head and the MoE router stay on AdamW; the router because Muon exacerbated early-training fluctuations. The stress test at constant learning rate compares the Qwen3.5 structure under AdamW, under Muon, and under Muon with the Gated Residual: the gate cuts both the frequency and the size of gradient-norm spikes, and a controlled toggle of GatedNorm with the optimizer fixed isolates the effect. The paper reads architecture, optimizer and stability as one design problem, which is its most transferable lesson.
Limits and open questions
Component ablations are at 25B-A3B on 480B to 560B tokens, below the production model. The headline comparison is base against base on pretraining benchmarks; no post-trained or agentic numbers appear. The 1/9-FLOPs figure follows from parameter and token ratios rather than reported total compute. And the host-memory n-gram design assumes serving hardware with enough host bandwidth to prefetch 51B parameters of tables without stalling.
FAQ
What is the Qwen3.8-Next architecture?
A 125B-parameter MoE with 6B active per token and 51B host-memory n-gram embedding parameters. Three Gated DeltaNet layers alternate with one attention layer, which becomes Qwen Sparse Attention at continued pretraining; the residual stream has four gated branches; Muon trains the 2D weights with embeddings, head and router on AdamW.
Why does Qwen3.8-Next use Gated DeltaNet instead of full attention?
Because a controlled ablation at 25B-A3B on identical tokens averaged 53.81 for the GDN hybrid against 49.87 for full attention and 51.15 for a sliding-window hybrid, with the largest gains on MATH, BBH and code, while one attention layer in four keeps exact retrieval.
How does Qwen Sparse Attention differ from DeepSeek V4’s compressed sparse attention?
QSA keeps full-resolution KV and uses a micro-block indexer to select which blocks to attend to, saving indexing and attention compute; DeepSeek V4 compresses groups of tokens into single KV entries before sparse selection, shrinking the cache itself. On Qwen’s model QSA raised RULER past 512K from 90.08 to 93.00.
What is the Gated Residual in Qwen3.8-Next compared with Kimi K3’s Attention Residuals?
Both widen the residual path. Qwen widens to four branches read through an elementwise gate with a group RMSNorm and finds the gate also stabilizes training; Kimi K3 instead lets each layer attend over earlier layers’ outputs in blocks of 12. DeepSeek V4’s mHC is a third variant with doubly stochastic mixing.