Research Papers
EN ZH

Language Models · Efficient AI · Open Models · Long Context

Qwen3.8-Next Architecture Explained: GDN Hybrid, QSA and Gated Residuals

Qwen3.8-Flash-Next is a 125B MoE with 6B active plus 51B host-memory n-gram tables; a 3:1 GDN-attention hybrid, sparse attention, gated residuals and Muon match the 397B Qwen3.7-Plus at 1/9 the FLOPs.

Alibaba Qwen Team · Paper published 2026-08-31 · 6 min read

Qwen3.8-Next Architecture Explained: GDN Hybrid, QSA and Gated Residuals

Quick answer

The Qwen team’s August 2026 paper is an architecture ablation report rather than a model card. Its subject, Qwen3.8-Flash-Next, has 125B total parameters, 6B activated per token, and a further 51B parameters of n-gram embedding tables that live in host memory and are prefetched, not in accelerator memory. On fourteen pretraining benchmarks the base model leads its 397B-A17B predecessor Qwen3.7-Plus on eight and trails on the other six by at most 2.6 points, using about a third of the activated parameters, a third of the training tokens and roughly a ninth of the training FLOPs. Four components carry the design: a layerwise hybrid of Gated DeltaNet (GDN) and global attention with one full-attention layer in every four; Qwen Sparse Attention (QSA), which replaces those full-attention layers at continued-pretraining time; a Gated Residual that widens the residual stream to four branches read through an elementwise gate; and the single n-gram embedding layer. Every candidate was judged on three axes at once: loss and downstream accuracy, cost in training, prefill and decode, and effect on optimal hyperparameters and stability.

Architecture at a glance

Token mixing. GDN layers compress the prefix into a fixed-size state at linear cost; the one attention layer in four keeps exact token-level retrieval, which the paper notes no finite-state memory reproduces. The ablation that justifies the split trains three 28-layer 25B-A3B checkpoints on the same 480B tokens: a full-attention Transformer, a sliding-window hybrid with a 128-token window, and the GDN hybrid. Nine-benchmark averages are 49.87, 51.15 and 53.81. The GDN hybrid leads on MATH (53.98 vs 49.40), BBH (68.72 vs 63.78) and MultiPL-E (47.48 vs 39.73).

Qwen Sparse Attention. At continued pretraining the full-attention layers become QSA. A lightweight indexer compresses the sequence into micro-block representations, scores them under a compressed causal mask and selects the top-k blocks, which expand into a micro-block sparse mask for the core attention. The point of micro-block granularity is that indexing cost itself falls with sequence length, which the paper contrasts with sparse designs whose indexer overhead stays non-negligible at long context. The kernel is released as FlashQLA.

Gated Residual. The residual stream is widened to four branches. Instead of the static or dynamic mixing matrices of hyper-connection designs, each sublayer reads the branches through an elementwise gate with a group RMSNorm and writes back; the paper’s ablation finds the mixing operator adds little once read and write are expressive enough, and that the gate is what supplies the rescaling that keeps training stable. Reads and writes are fused into single kernels so the widened stream is traversed once per block in each direction.

N-gram embedding. A single layer of n-gram embedding tables, conditioned on local context rather than token identity alone, adds 51B parameters with negligible per-token FLOPs because the tables are prefetched from host memory. Enlarging the n-gram vocabulary lowers loss monotonically, but downstream accuracy saturates, one of the paper’s examples of loss and accuracy disagreeing.

Optimizer and stability. Muon is the main optimizer on the two-dimensional weights that act as linear maps: attention and GDN projections, expert MLPs, and the n-gram key and value projections. Embeddings, the output head and the MoE router stay on AdamW, the router because Muon destabilized it early in training. A stress test at constant learning rate on the 25B-A3B model compares the Qwen3.5 structure under AdamW, the same under Muon, and Muon with the Gated Residual: the gate reduces both the frequency and the size of gradient-norm spikes, and a separate ablation isolates the effect of GatedNorm with the optimizer held fixed.

Key results

  • Base model vs predecessor (Table 11): Qwen3.8-Flash-Next-Base against Qwen3.7-Plus-Base, MMLU 90.36 vs 90.43, MMLU-Pro 73.23 vs 70.90, SuperGPQA 51.36 vs 48.42, BBH 90.87 vs 89.41, GPQA 51.42 vs 51.52, GSM8K 93.29 vs 92.95, MATH 72.78 vs 74.38, EvalPlus 78.76 vs 78.06, MultiPL-E 79.09 vs 81.68, SWEBench-Pretrain 50.99 vs 49.24, MGSM 89.33 vs 85.42, MMMLU 84.86 vs 84.53, INCLUDE 78.40 vs 78.90. The dense Qwen3.8-27B-Base trails both on most rows.
  • Hybrid ablation (Table 1): full attention 49.87, SWA hybrid 51.15, GDN hybrid 53.81 average over nine benchmarks at 25B-A3B.
  • QSA vs full attention on the production model (Table 2): short-context average 75.9 with full attention, 76.8 with QSA; MMLU-Pro 72.9 vs 73.7, MATH 69.8 vs 71.6, EvalPlus 70.8 vs 72.3.
  • Long context (Table 3): RULER 512K to 1M 90.08 with full attention, 93.00 with QSA; 8-needle MRCR at 512K 30.66 vs 40.53 and at 1M 20.71 vs 26.44; RULER at 128K essentially tied at 99.8.
  • Speculative decoding unaffected (Table 4): mean MTP accepted length 4.06 with full attention, 4.07 with QSA under four-step drafting.
  • Efficiency claim: about 1/3 the activated parameters, 1/3 the training tokens and roughly 1/9 the training FLOPs of Qwen3.7-Plus for the results above.

Limits and open questions

The headline comparison is base model against base model on pretraining benchmarks; there are no post-trained or agentic results in this paper. The 1/9-FLOPs figure follows from the parameter and token ratios rather than from a reported total compute for either run. The hybrid ablation is at 25B-A3B on 480B tokens, and the residual ablation at 560B tokens, so component conclusions are drawn below the production scale. Off-accelerator n-gram tables shift 51B parameters to host memory and prefetch, which the paper says costs negligible latency but which depends on the serving system’s host bandwidth. How this design compares with DeepSeek-V4’s compressed attention and Kimi K3’s delta attention is in Qwen3.8-Next Architecture Explained.

This paper is one of the sources behind Kimi K3 Architecture Explained, Muon vs AdamW.

FAQ

What is the Qwen3.8-Next architecture?

A 125B-parameter sparse MoE with 6B active per token and 51B of host-memory n-gram embedding tables. Token mixing alternates three Gated DeltaNet layers with one attention layer, and Qwen Sparse Attention replaces the attention layers at continued pretraining. The residual stream has four branches read through an elementwise gate, and most matrices are trained with Muon.

How does Qwen3.8-Flash-Next compare with Qwen3.7-Plus?

The base model leads the 397B-A17B Qwen3.7-Plus on 8 of 14 pretraining benchmarks and trails by at most 2.6 points on the rest, for example MMLU-Pro 73.23 vs 70.90 and MultiPL-E 79.09 vs 81.68, with about a third of the activated parameters and tokens and roughly a ninth of the training FLOPs.

Does Qwen Sparse Attention lose accuracy compared with full attention?

No in the reported settings. Short-context averages rise from 75.9 to 76.8, RULER at 512K to 1M rises from 90.08 to 93.00, and 8-needle MRCR at 1M rises from 20.71 to 26.44, while MTP accepted length is unchanged at about 4.07.

Why does Qwen3.8-Next use Gated DeltaNet instead of sliding-window attention?

In a controlled ablation at 25B-A3B on the same tokens, the GDN hybrid averaged 53.81 against 51.15 for a sliding-window hybrid and 49.87 for full attention, with the largest gains on MATH, BBH and MultiPL-E, while keeping one full-attention layer in four for exact retrieval.