Sequence Modeling · Transformers · Efficient AI · Long Context
Mamba vs Transformer: Throughput, Recall and What Scale Changes
Mamba swaps attention's quadratic compute and growing KV cache for a fixed-size selective state: 5x decode throughput and parity with 2x-larger Transformers at 3B, but exact recall stays with attention.
What each architecture keeps in memory
The two designs answer one question differently: what does the model remember about the tokens it has already seen?
A Transformer keeps everything. Every earlier token leaves a key and a value in cache, and each new token scores all of them. That is why the path between any two positions is constant length, why the model can retrieve an exact token from 50,000 positions back, and why compute per token grows with context length while the cache grows without bound. FlashAttention made this 2 to 4x faster by never materializing the attention matrix in GPU memory, but it did not change the quadratic FLOP count; it changed the memory traffic.
Mamba keeps a fixed-size summary. A selective state space layer carries a hidden state forward one token at a time and lets the input decide, through input-dependent step size and projections, what to write into the state and what to let decay. Because the state does not grow, decoding costs the same at token 1,000,000 as at token 100, and there is no KV cache to page. The price is that the state is a lossy compression: what the model did not choose to keep at the time is gone.
The 2026 study On Subquadratic Architectures puts Mamba-2 next to xLSTM and Gated DeltaNet in one gating framework and shows that within the subquadratic family, which gate you use decides what the state can compute. That matters for this comparison because “Mamba” in 2026 is one point in a design space, not the whole alternative to attention.
Key numbers
| Measurement | Transformer | Mamba or SSM | Setting and source | Same harness? |
|---|---|---|---|---|
| Inference throughput, autoregressive decode | 1x | 5x | Mamba paper, same parameter count | Yes |
| Per-token compute vs context length | Grows with length (quadratic total) | Constant | Design | Yes, by construction |
| State carried between tokens | KV cache, grows linearly | Fixed-size hidden state | Design | Yes |
| Language modeling at 3B | Baseline Transformer | Mamba-3B beats same-size, matches 2x-size | Pretraining perplexity and downstream evals, Mamba paper | Yes |
| Longest length where quality still improves | Not reported | Up to 1M-length sequences on real data | Mamba paper | No |
| Longest context beating chance on Path-X | 16K, 61.4%, with FlashAttention | Not reported for Mamba | FlashAttention paper | No |
| Exact attention speedup from IO-awareness | 2 to 4x, GPT-2 1K: 3x end to end | n/a | FlashAttention | n/a |
| State tracking (parity, S3) at length 2048 | TC0 ceiling in the paper’s framing | Mamba-2 never above 0.352; xLSTM[1:1] 1.000 | Trained at 128, subquadratic study | Yes, within the SSM family |
| Counting (majority) extrapolated to 2048 | Not run | Mamba-2 0.241; Gated DeltaNet 0.268; xLSTM[1:0] 0.763 | Same study | Yes, within the SSM family |
| Time-series GIFT-Eval at 10M params | Not run | Mamba-2 MASE 0.767 / CRPS 0.525; xLSTM[3:1] 0.733 / 0.508 | Same study; converges by 80M | Yes, within the SSM family |
| Largest scale with head-to-head evidence | Frontier models are Transformers or hybrids | About 3B (Mamba), 400M (2026 study) | Both papers | No |
The first four rows are the comparison people mean when they type “mamba vs transformer”, and all come from the Mamba paper’s own controlled runs. The bottom rows compare SSM variants with each other, not with attention; they are here because they explain where Mamba’s fixed state breaks, which is the same reason it loses to attention on recall.
Where the 5x comes from, and where it does not
The throughput gain is a decode-time result. During generation a Transformer re-reads a growing KV cache for every new token, so it becomes memory-bound as context lengthens; Mamba updates a constant-size state, so its per-token cost is flat. The Mamba paper reports 5x higher inference throughput at matched size, and quality that keeps improving out to million-token sequences, the regime where dense attention is impractical.
Training is a different story. Time-invariant SSMs could be computed as one global convolution, which is fast. Making the parameters input-dependent breaks that, and Mamba’s answer is a hardware-aware parallel scan that keeps the expanded state in SRAM. That is the same class of trick FlashAttention used for attention: the algorithmic win only shows up with a kernel that respects the GPU memory hierarchy. Neither architecture’s headline speed transfers to a framework that cannot fuse the operation.
Recall is the structural trade
Attention can address any earlier token because every token is still there. A recurrent state cannot; it holds whatever the selection mechanism chose to keep. The Mamba paper names this directly: tasks needing exact recall of an arbitrary earlier token, copying and precise retrieval, can favor explicit attention. The subquadratic study makes the mechanism concrete on synthetic tasks. Models trained at length 128 and tested at 2048 show Mamba-2 collapsing on counting (majority accuracy 0.241) and never solving parity or permutation composition (S3), while an xLSTM variant with separate, exponential input and forget gates extrapolates to 0.763 on majority and 1.000 on parity and S3. The gating math decides whether a fixed state can keep a running tally past the training length. Mamba-2’s tied gates cannot.
This is why the field went hybrid rather than pure SSM. Interleaving a few attention layers into a Mamba stack restores an exact-recall path at a fraction of the KV cache; the Mamba paper anticipated it and most 2025 and 2026 production “Mamba” models are built that way.
When to use which
- Long-context retrieval, needle tasks, code with far references: Transformer, or a hybrid with attention layers. Exact recall is the one thing a fixed state provably cannot guarantee.
- Throughput-bound decode, streaming, on-device, models up to a few billion parameters: Mamba-style layers. The 5x decode number and the 3B parity result are the evidence, and they are strongest at exactly this scale.
- Contexts under 8K on a dense Transformer: the cost you are trying to avoid is small. FlashAttention-class kernels already remove most of the memory penalty, and the Mamba advantage is hard to observe.
- Choosing among subquadratic layers in 2026: do not assume Mamba. The subquadratic study finds xLSTM variants ahead on code pretraining (by 0.90 to 1.81 HumanEval points at 400M) and on state tracking, with the gap fading at larger time-series scales. Test the gating variant on your task’s primitive.
Limits and open questions
The controlled Mamba-versus-Transformer evidence tops out near 3B parameters, and the 2026 subquadratic comparison at 400M for code and 80M for time series, where its own margins already shrink or flip. Whether a pure selective SSM matches attention at tens of billions of parameters is untested in these papers; the practical answer the field chose is hybrids, which is an admission that the recall gap is real. The Transformer side of the table is also older than it looks: the 2017 paper measured translation, not language modeling, so the “Transformer baseline” in every row is whichever Transformer the SSM paper trained, under the SSM paper’s recipe. And no paper here reports a matched, long-context, exact-retrieval benchmark for Mamba against a FlashAttention Transformer at equal wall-clock budget, which is the number a deployment decision actually needs.
FAQ
Is Mamba faster than a Transformer?
At decode time, yes: the Mamba paper reports 5x higher inference throughput at matched parameter count, because Mamba updates a fixed-size state instead of re-reading a growing KV cache. For training and short contexts the gap is small, and a Transformer with FlashAttention already recovers 2 to 4x over naive attention.
Does Mamba match Transformer quality at the same size?
At 3B, the Mamba paper reports Mamba-3B beating same-size Transformers and matching Transformers twice its size on pretraining perplexity and downstream evaluations. There is no comparable controlled result at tens of billions of parameters, where production models are Transformers or hybrids.
Why do Transformers beat Mamba on copying and exact recall?
Because attention keeps every earlier token addressable in the KV cache, while Mamba compresses history into a fixed-size state and can only retrieve what its selection mechanism chose to keep. The 2026 subquadratic study shows the same limit on synthetic tasks: Mamba-2 stays at or below 0.352 on parity and S3 at length 2048 while a differently gated xLSTM reaches 1.000.
Is Mamba still the best subquadratic architecture compared with xLSTM and Gated DeltaNet?
Not across the board. In the 2026 subquadratic study, xLSTM variants lead Mamba-2 on code pretraining at 400M, on counting and state-tracking extrapolation, and on small-scale time-series forecasting, though Mamba-2 takes CRPS back by 0.005 at 80M parameters. Which gate the layer uses decides the result more than the Mamba name.