Long Context · Efficient AI · Transformers
Sparse Attention vs Full Attention: Where the Speedups Come From
Sparse attention buys 7x to 28x less attention compute at 1M tokens for roughly half a point on long-context suites, but the wall-clock gain is far smaller than the FLOPs gain and most of it is invisible at 32K.
Four things people mean by “sparse attention”
Full attention lets every query token score every earlier key, so cost grows with the square of context length. Everything called sparse attention cuts the number of keys each query touches, but the recent papers do it in three different places, and the fourth method in this comparison is not sparse at all.
Block selection with a learned indexer. MiniMax Sparse Attention (MSA) puts a lightweight Index Branch in front of each attention layer. It scores key-value blocks of 128 tokens, keeps the top 16 plus the local block, and runs exact softmax attention over those 2,048 tokens only. The indexer is trained with a KL loss to imitate where full attention puts its mass, and its gradients are detached from the main path.
Head specialization. Full Attention Strikes Back (RTPurbo) starts from a trained dense model and observes that only a minority of heads reach far back in the context. It keeps about 15% of heads as full-context retrieval heads with a 16-dimensional indexer and converts the other 85% to an 8,192-token sliding window with 4 sink tokens. Two adaptation stages of roughly 600 steps each do the conversion.
KV offloading with lookahead. FlashMemory-DeepSeek-V4 does not change which tokens attention can see so much as which are resident in GPU memory. A dual-encoder Neural Memory Indexer predicts which KV chunks the next span will query and keeps only those on the GPU, so the decoding KV cache shrinks to 13.5% of the full-context baseline on average.
Exact attention with less memory traffic. FlashAttention computes the same numbers as full attention. Its 2 to 4x speedup comes from tiling the computation so the N by N attention matrix never lands in HBM. It is the correct baseline for “full attention”, because a sparse method that beats naive attention but not FlashAttention has not beaten anything a production stack runs.
Key numbers
| Measurement | Full attention | Sparse variant | Model, setting, source | Same harness? |
|---|---|---|---|---|
| Per-token attention FLOPs at 1M context | 1x | 1/28.4 | 109B multimodal model, MSA paper | Yes |
| Prefill / decode wall clock at 1M, H800 | 1x / 1x | 14.2x / 7.6x faster | Same model, co-designed kernel, MSA | Yes |
| KV tokens attended per query group | All | 2,048 (16 blocks of 128) | MSA design | Yes |
| HELMET-128K overall | 46.53 | 45.93 | MSA-CPT after 140B long-context tokens | Yes |
| RULER-128K overall | 72.00 | 72.12 | Same run, MSA | Yes |
| MMLU / GSM8K under a 3T-token budget | 67.0 / 76.2 | 67.2 / 77.7 | MSA-PT trained sparse from scratch | Yes |
| Prefill speedup at 1M / at 32K | 1x | 9.36x / 2.83x | Qwen3-Coder-30B-A3B, RTPurbo | Yes |
| Decode speedup at 1M | 1x | about 2.01x | Same, RTPurbo | Yes |
| LongBench average | 53.80 | 54.24 | RTPurbo vs its own dense teacher | Yes |
| RULER at 64K | 86.23 | 85.49 | RTPurbo | Yes |
| AIME24/25 accuracy | 86.67 | 86.67 | Qwen3-30B-A3B-Think, RTPurbo | Yes |
| Heads with full context | 100% | 15% | RTPurbo design | n/a |
| Decode KV cache resident on GPU | 100% | 13.5% average, over 90% cut at 500K | FlashMemory-DeepSeek-V4 on LongBench-v2, LongMemEval, RULER | Yes |
| Accuracy change from KV offloading | baseline | +0.6 points average | Same suites, FlashMemory-DeepSeek-V4 | Yes |
| Exact attention speedup, GPT-2 at 1K | naive 1x | 3x end to end | FlashAttention, same outputs | Yes, exact |
Every row compares a sparse method with the dense model it was derived from or trained alongside, inside one paper. Rows from different papers are not comparable with each other: MSA’s 109B model, RTPurbo’s Qwen3-Coder-30B-A3B and FlashMemory’s DeepSeek-V4 backbone differ in size, training data and kernels.
FLOPs are not latency
The most quoted MSA number, 28.4x fewer attention FLOPs at 1M tokens, becomes 14.2x on prefill and 7.6x on decode once measured on H800s. The paper says why: sparse attention adds index construction, top-k selection, gathering the selected blocks, reverse indexing and load balancing across GPUs. None of that shows up in a FLOP count. RTPurbo tells the same story from the other end. Its 9.36x prefill gain at 1M tokens shrinks to 2.83x at 32K, because the quadratic term that sparsity removes is small at 32K and the fixed overhead of the indexer and the retrieval heads’ dense prefill is not. FlashAttention is the reference point here: it gets 2 to 4x on exact attention purely by reducing memory traffic, so at the context lengths most deployments run today, a good dense kernel and a sparse method land in the same range.
The practical rule: quote sparse speedups with the context length attached. A “10x faster” claim at 1M tokens says almost nothing about a 16K chat workload.
Where the quality goes
Aggregate scores move by less than a point in every paper, and in two cases the sparse model edges out its teacher: RTPurbo’s LongBench 54.24 against 53.80, MSA-PT’s GSM8K 77.7 against 76.2. The losses are where you would expect a fixed retrieval budget to hurt. MSA’s HELMET breakdown shows Rerank and RAG dropping 2.10 points while multi-key and multi-value retrieval improve 2.24, a real trade rather than noise. RTPurbo gives back 0.74 points on RULER at 64K while matching AIME exactly. FlashMemory reports +0.6 on average but warns that an indexer that misses the one chunk holding the answer produces a failure that averages hide.
The pattern across all three: tasks that need one or a few precise lookups survive sparsification, and sometimes improve because the model ignores distractors. Tasks that integrate many weak pieces of evidence spread across the context are where a 2,048-token or 15%-of-heads budget can drop something without any visible signal.
When to use which
- Contexts routinely above 128K, you control the serving stack: sparse attention, with the kernel as part of the deliverable. MSA-style block selection if you can pretrain or continue-pretrain; RTPurbo-style head conversion if you must keep an existing dense checkpoint and can afford roughly 1,200 adaptation steps.
- Memory-bound decode at 500K, not compute-bound prefill: KV offloading with a lookahead indexer attacks the right bottleneck. It keeps full attention semantics and cuts resident cache by about 7x.
- Contexts under 32K: full attention with FlashAttention or a successor kernel. The sparse gain is under 3x and you keep exactness, simpler serving and no indexer to train.
- Reasoning-heavy workloads: the AIME tie is reassuring, but it is one model family. Measure before and after on your own long-chain tasks; the failure mode is silent.
Limits and open questions
All three sparse results live inside one lab’s model and kernel stack: MSA in MiniMax’s 109B model, RTPurbo on Qwen3 variants, FlashMemory on a DeepSeek-V4 backbone. Portability to other sizes, GPUs and serving systems is asserted, not measured. RTPurbo’s head split assumes head roles are stable under domain shift. None of the papers publishes an adversarial evidence-integration test that would expose the fixed-budget failure. And the comparison you actually want, the same dense model converted three ways and evaluated on one harness, does not exist yet.
FAQ
Is sparse attention faster than full attention at 32K context?
Only modestly. RTPurbo reports a 2.83x prefill speedup at 32K on Qwen3-Coder-30B-A3B against 9.36x at 1M, and FlashAttention already gives exact attention a 2 to 4x speedup by reducing memory traffic. The sparse advantage becomes large only past roughly 128K tokens.
Does sparse attention lose accuracy compared with full attention?
By less than a point on aggregate long-context suites: MSA scores 45.93 against 46.53 on HELMET-128K and 72.12 against 72.00 on RULER-128K; RTPurbo scores 85.49 against 86.23 on RULER at 64K and 54.24 against 53.80 on LongBench. Losses concentrate on tasks that combine many scattered pieces of evidence, such as MSA’s 2.10-point drop on Rerank and RAG.
What is the difference between FlashAttention and sparse attention?
FlashAttention is exact: it produces the same outputs as full attention and is faster only because it avoids writing the attention matrix to GPU memory. Sparse attention changes the computation so each query attends to a subset of keys, which cuts FLOPs by up to 28.4x at 1M tokens but can change outputs.
Can you convert a full attention model to sparse attention without retraining from scratch?
Yes. RTPurbo converts Qwen3-Coder-30B-A3B with two adaptation stages of about 600 steps each, keeping 15% of heads at full context and switching the rest to an 8,192-token sliding window, and matches or slightly beats the dense model on LongBench and AIME. MSA-CPT similarly continues pretraining a dense checkpoint with sparse attention.