Research Papers
EN ZH

Language Models · Long Context · Open Models · Efficient AI

DeepSeek-V4 Explained: Compressed Sparse Attention and 1M Context at 10% KV

DeepSeek-V4 ships two MoE models, 1.6T/49B and 284B/13B, with compressed sparse plus heavily compressed attention that cuts 1M-context KV cache to 10% of V3.2, mHC residuals and Muon.

DeepSeek · Paper published 2026-04-26 · 6 min read

DeepSeek-V4 Explained: Compressed Sparse Attention and 1M Context at 10% KV

Quick answer

DeepSeek-V4 is a preview series of two mixture-of-experts models released in April 2026: DeepSeek-V4-Pro with 1.6T total and 49B activated parameters, and DeepSeek-V4-Flash with 284B total and 13B activated. Both natively support a one-million-token context. Three changes carry the report. A hybrid attention stack of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) that, at 1M tokens, brings V4-Pro to 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2, and V4-Flash to 10% and 7%. Manifold-Constrained Hyper-Connections (mHC), a widened residual stream whose mixing matrix is projected onto doubly stochastic matrices so deep stacks stay numerically stable. And the Muon optimizer for most modules, adopted for faster convergence and training stability. Pretraining used more than 32T tokens; V4-Pro-Max, the maximum-reasoning-effort mode, tops the open models in the report’s own table on LiveCodeBench, Codeforces and Apex Shortlist while trailing Gemini-3.1-Pro on knowledge.

The attention stack, and why it is two mechanisms

Long context makes attention the dominant cost, so V4 attacks it twice and interleaves the results. CSA first compresses the key-value cache of every group of m consecutive tokens into one entry using learned compression weights and positional biases, then runs DeepSeek Sparse Attention on the compressed entries: a Lightning Indexer scores them in a low-rank way and each query attends only to the top-k compressed entries. A small sliding-window branch of uncompressed recent tokens is added back for local detail, and an attention-sink term lets each head’s scores sum to less than one. HCA compresses more aggressively, consolidating a much larger group of tokens per entry, and keeps the same sliding window and shared-KV multi-query layout. Layers alternate between the two. The KV entries themselves are stored with BF16 for the rotary dimensions and FP8 for the rest, and routed-expert weights are held in FP4.

The efficiency numbers follow from the design rather than from a kernel alone: compression shrinks the cache by the group size, sparsity shrinks the compute per query to top-k, and HCA layers shrink both further. Figure 1 plots the result against V3.2 across context lengths.

Manifold-constrained residuals and Muon at scale

Hyper-connections widen the residual stream by a factor and let layers read and write through learned input, output and residual mixing maps. DeepSeek found the naive version numerically unstable when stacked. mHC constrains the residual mixing matrix to the Birkhoff polytope of doubly stochastic matrices via Sinkhorn-Knopp normalization, so its spectral norm is at most one and the map is non-expansive in both forward and backward passes; input and output maps are bounded with sigmoids. The set is closed under multiplication, which is what keeps a deep stack of such blocks stable.

Muon is used for the majority of weight matrices, with AdamW kept for embeddings, the prediction head, mHC’s static biases and gates, and RMSNorm weights. The report adds a hybrid Newton-Schulz schedule and a specific guard against exploding attention logits, and Section 3.4 describes a distributed Muon implementation. This is the largest disclosed Muon pretraining run to date, and V4’s adoption was followed by Kimi K3 and Qwen3.8-Next.

Key results

  • Efficiency at 1M tokens (Figure 1): V4-Pro at 27% of V3.2’s single-token FLOPs and 10% of its KV cache; V4-Flash at 10% and 7%. V3.2 activates 37B parameters against V4-Flash’s 13B and V4-Pro’s 49B.
  • Pretraining: 32T tokens for Flash, 33T for Pro; 128K vocabulary inherited from V3 with new context-construction tokens; long-document curation emphasizing scientific papers and technical reports.
  • V4-Pro-Max against closed models (Table 6): LiveCodeBench 93.5 (best in table; Gemini-3.1-Pro 91.7, Opus-4.6 88.8), Codeforces rating 3206 (best; GPT-5.4 3168), Apex Shortlist 90.2 (best; Gemini 89.1), HMMT Feb 2026 95.2 (GPT-5.4 97.7), IMOAnswerBench 89.8 (GPT-5.4 91.4), GPQA Diamond 90.1 (Gemini 94.3), HLE 37.7 (Gemini 44.4), MMLU-Pro 87.5 (Gemini 91.0), SimpleQA-Verified 57.9 (Gemini 75.6, Opus 46.2).
  • Agentic (Table 6): SWE Verified 80.6 (Opus-4.6 80.8, Gemini 80.6), SWE Pro 55.4 (K2.6 58.6), Terminal Bench 2.0 67.9 (GPT-5.4 75.1; about 72.0 on the Verified subset), BrowseComp 83.4 (Gemini 85.9), HLE with tools 48.2 (K2.6 54.0), GDPval-AA Elo 1554 (GPT-5.4 1674), MCPAtlas 73.6 (Opus 73.8).
  • Long context: MRCR 1M 83.5 and CorpusQA 1M 62.0, against Opus-4.6’s 92.9 and 71.7.
  • Reasoning effort matters (Table 7): V4-Pro Non-Think, High and Max score 72.9, 89.1 and 90.1 on GPQA Diamond; 7.7, 34.5 and 37.7 on HLE; 73.6, 79.4 and 80.6 on SWE Verified. V4-Flash-Max reaches 88.1 GPQA, 79.0 SWE Verified, 56.9 Terminal Bench and 91.6 LiveCodeBench at 13B activated.

Limits and open questions

The report calls itself a preview. Every comparison in Table 6 runs closed models under DeepSeek’s own harness with 500 interaction steps and 512K context, and the authors note GPT-5.4 could not be evaluated on the coding-agent tasks because its API failed on many queries. The knowledge gap to Gemini-3.1-Pro on SimpleQA, HLE and MMLU-Pro is real and acknowledged. Long-context scores at 1M trail Opus-4.6 by 9 to 10 points, so the efficiency claim is about cost per token at 1M, not parity of recall there. The compression group sizes and top-k are documented in the released inference code rather than fully in the text. And the efficiency figures are FP8-equivalent FLOP estimates and cache sizes, not measured latency on a stated serving stack.

This paper is one of the sources behind DeepSeek V4 Architecture Explained, Kimi K3 Architecture Explained, Muon vs AdamW, Nemotron 3 Architecture Explained, Qwen3.8-Next Architecture Explained.

FAQ

What is the DeepSeek-V4 architecture?

A DeepSeekMoE feed-forward stack with a hybrid attention design that interleaves Compressed Sparse Attention (compress groups of tokens into single KV entries, then attend sparsely to the top-k via a Lightning Indexer) and Heavily Compressed Attention (much larger groups), both with a small sliding-window branch and an attention sink. Residual connections use manifold-constrained hyper-connections, and most matrices are trained with Muon.

How much more efficient is DeepSeek-V4 than V3.2 at 1M context?

V4-Pro needs 27% of V3.2’s single-token inference FLOPs and 10% of its KV cache at a one-million-token context; V4-Flash needs 10% and 7%. These are FP8-equivalent estimates from the report’s Figure 1, not measured latency.

How does DeepSeek-V4-Pro-Max compare with Claude Opus 4.6 and Gemini 3.1 Pro?

In the report’s own table it leads on LiveCodeBench (93.5), Codeforces (3206) and Apex Shortlist (90.2), matches Opus on SWE Verified (80.6 vs 80.8), and trails Gemini on knowledge: SimpleQA-Verified 57.9 vs 75.6, HLE 37.7 vs 44.4, GPQA Diamond 90.1 vs 94.3. Baselines were run in DeepSeek’s harness.

What are the differences between DeepSeek-V4-Pro and V4-Flash?

Pro has 1.6T total and 49B activated parameters; Flash has 284B total and 13B activated. At Max effort, Flash scores 88.1 GPQA Diamond, 79.0 SWE Verified and 91.6 LiveCodeBench against Pro’s 90.1, 80.6 and 93.5, while using 10% of V3.2’s FLOPs at 1M context versus Pro’s 27%.