Language Models · Long Context · Efficient AI · Open Models
DeepSeek V4 Architecture Explained: CSA, HCA, mHC and Muon
DeepSeek V4 keeps V3's MoE and multi-token prediction, rebuilds attention as a compress-then-sparsify hybrid, widens the residual stream under a doubly stochastic constraint, and trains most matrices with Muon.
Architecture at a glance
Start from what did not change. DeepSeek-V3 contributed the DeepSeekMoE feed-forward design with fine-grained routed experts plus shared experts, auxiliary-loss-free load balancing, and multi-token prediction as a training objective and a speculative-decoding draft. V4 inherits all of that, the 128K-token vocabulary, the token-splitting and fill-in-the-middle data strategies, and the FP8-style low-precision training stance, now extended to FP4 for routed-expert weights. Two model sizes ship: V4-Pro at 1.6T total and 49B activated parameters, V4-Flash at 284B and 13B. V3.2 activated 37B.
Three things are new. Attention layers alternate between two designs, Compressed Sparse Attention and Heavily Compressed Attention, both built to make one-million-token context cheap. Residual connections become manifold-constrained hyper-connections, a widened residual stream that stays stable when stacked deep. And the optimizer for most weight matrices is Muon rather than AdamW. The technical report presents these as the reasons V4 can routinely run 1M-token contexts.
Key numbers
| Component | V3 / V3.2 | V4 | Source | Same harness? |
|---|---|---|---|---|
| Total / activated parameters | V3.2: 37B activated | Pro 1.6T / 49B; Flash 284B / 13B | V4 report | n/a |
| Pretraining tokens | V3: 14.8T | Flash 32T, Pro 33T | Reports | n/a |
| Attention | MLA, then DSA sparse selection in V3.2 | CSA (compress m tokens per entry, then DSA top-k) alternating with HCA (larger groups), each with a sliding window and attention sink | V4 report, Section 2.3 | n/a |
| Single-token inference FLOPs at 1M context | 1x (V3.2) | Pro 27%, Flash 10% | V4 Figure 1, FP8-equivalent estimate | Yes |
| KV cache at 1M context | 1x (V3.2) | Pro 10%, Flash 7% | V4 Figure 1 | Yes |
| KV storage precision | FP8 latent (V3.2) | BF16 for RoPE dimensions, FP8 for the rest | V4 report | n/a |
| Residual stream | standard pre-norm residual | mHC: widened stream, residual mixing projected onto doubly stochastic matrices via Sinkhorn-Knopp | V4 Section 2.2 | n/a |
| Optimizer | AdamW | Muon for most matrices; AdamW for embeddings, head, mHC static biases and gates, RMSNorm | V4 Section 2.4 | n/a |
| Native context | 128K | 1M | Reports | n/a |
| MRCR 1M / CorpusQA 1M | n/a | 83.5 / 62.0 (Opus-4.6: 92.9 / 71.7) | V4 Table 6 | Yes, DeepSeek harness |
| GPQA Diamond, Non-Think / High / Max | n/a | Pro 72.9 / 89.1 / 90.1; Flash 71.2 / 87.4 / 88.1 | V4 Table 7 | Yes |
| SWE Verified at Max | n/a | Pro 80.6; Flash 79.0 | V4 Table 7 | Yes |
The efficiency rows are estimates from the report’s own figure, not measured latency; the accuracy rows use DeepSeek’s harness with 500 steps and 512K context.
How compress-then-sparsify attention works
CSA does two reductions in sequence. First, compression: every group of m consecutive tokens’ keys and values is folded into one KV entry, using trainable compression weights, a softmax over the group, and learnable positional biases. Indexes overlap between the key and value paths, so the sequence length seen by attention shrinks by the group size. Second, sparsity: DeepSeek Sparse Attention, the mechanism V3.2 introduced, runs on the compressed entries. A Lightning Indexer builds low-rank indexer queries and compressed indexer keys, scores them, and each query attends only to its top-k compressed entries. Two details keep local fidelity: a small sliding-window branch of uncompressed recent tokens, and an attention-sink term that lets a head’s attention scores sum to less than one so it can effectively attend to nothing.
HCA is the same idea with a much larger group per entry and no sparse selection stage, plus the same sliding window and shared-KV multi-query layout. Interleaving CSA and HCA layers is what produces the 10% KV figure: CSA layers cut cache by the compression ratio and compute by top-k, HCA layers cut cache further.
The contrast with MiniMax Sparse Attention is instructive. MSA keeps full-resolution KV blocks of 128 tokens and selects 16 of them per query, a fixed 2,048-token budget, so its KV cache is unchanged and its savings are compute. V4 shrinks the cache itself before selecting, which is why its KV number is the headline. The trade is fidelity within a group: a compressed entry cannot be un-mixed at query time. FlashMemory-style lookahead offloading, developed on a V4 backbone, attacks the residual cache from yet another angle by predicting which entries to keep resident.
Why mHC and why Muon
Hyper-connections widen the residual stream by a factor and let each layer read from and write to the wider stream through learned maps. The naive version was numerically unstable when stacked, so V4 constrains the residual mixing matrix to the Birkhoff polytope of doubly stochastic matrices using Sinkhorn-Knopp normalization. Doubly stochastic matrices have spectral norm at most one and are closed under multiplication, so a deep stack of residual mixes is non-expansive in both directions. Input and output maps are bounded with sigmoids. The report treats this as what made hyper-connections usable at 1.6T scale.
Muon replaces AdamW for the majority of modules. DeepSeek cites faster convergence and improved training stability, adds a hybrid Newton-Schulz schedule and a guard against exploding attention logits, and keeps AdamW where Muon does not apply: embeddings, the prediction head, mHC’s static biases and gates, and RMSNorm weights. This is the largest disclosed Muon pretraining run, and Kimi K3 and Qwen3.8-Next followed with per-head and router-excluded variants.
Limits and open questions
The compression group sizes and top-k are specified in the released inference code rather than fully in the paper text. Efficiency is reported as FP8-equivalent FLOPs and cache bytes, and FP4 expert weights do not yet run faster than FP8 on current hardware. Long-context accuracy at 1M trails Claude Opus 4.6 by 9 to 10 points on MRCR and CorpusQA, so the architecture buys cost, not parity, at that length. There is no ablation isolating CSA from HCA, mHC from plain residuals, or Muon from AdamW at V4 scale; the report describes the design and the outcome, not the counterfactuals.
FAQ
What is the DeepSeek V4 architecture?
A DeepSeekMoE model inherited from V3, with attention rebuilt as alternating Compressed Sparse Attention (compress groups of tokens into single KV entries, then attend sparsely to the top-k via a Lightning Indexer) and Heavily Compressed Attention (much larger groups), residuals replaced by manifold-constrained hyper-connections, and Muon as the main optimizer. Pro has 1.6T total and 49B active parameters; Flash 284B and 13B.
How is DeepSeek V4’s architecture different from V3?
V3 used MLA attention with a 128K context and AdamW. V4 keeps the MoE and multi-token prediction but adds compress-then-sparsify attention for a native 1M context, mHC residuals for depth stability, FP4 routed experts, and Muon for most matrices. At 1M tokens V4-Pro needs 27% of V3.2’s FLOPs and 10% of its KV cache.
How does DeepSeek V4’s compressed sparse attention compare with MiniMax sparse attention?
MSA selects 16 full-resolution 128-token blocks per query and keeps the KV cache unchanged, so its savings are compute. V4’s CSA compresses groups of tokens into single entries before sparse selection, shrinking the cache itself, and HCA layers compress further; that is why V4’s headline is a 10% KV cache rather than a FLOPs multiple.
Which parts of DeepSeek V4 are trained with Muon and which with AdamW?
Most weight matrices use Muon with a hybrid Newton-Schulz schedule. Embeddings, the prediction head, the static biases and gates of the mHC modules, and RMSNorm weights stay on AdamW.