Mamba: A Serious Attempt to Challenge Attention on Long Sequences

What problem it solves

Transformers dominate foundation models, but attention is costly on long sequences. Many efficient alternatives scale better but fail to match attention on language and other discrete modalities. Mamba targets that gap: build a sequence model that scales linearly while retaining enough content-based reasoning to compete with Transformers.

The core method

Mamba improves structured state space models by making their parameters depend on the input. That selectivity lets the model decide what information to propagate or forget at each token. Because this breaks some efficient convolution tricks, the authors design a hardware-aware parallel algorithm in recurrent mode and package the result into a simplified architecture without attention or even standard MLP blocks.

Key results

The paper reports fast inference, linear scaling in sequence length, and performance that improves up to million-length sequences. As a general backbone, Mamba reaches strong results across language, audio, and genomics. In language modeling, the Mamba-3B model outperforms same-size Transformers and matches Transformers twice its size in several evaluations.

Why it matters

Mamba gave the field a credible non-attention backbone at a time when long context was becoming central. Even when Transformers remain dominant, Mamba changed the architecture conversation: recurrence and state space models were no longer merely old ideas, but hardware-aware candidates for modern foundation models.

Limits and open questions

Mamba is not a drop-in replacement for every Transformer workload. Ecosystem support, scaling behavior at the largest frontier sizes, training recipes, and hybrid architectures remain active questions. The core tradeoff is also still under debate: when does selective memory beat explicit pairwise attention, and when does the lack of direct token-token comparison hurt?

One line: Mamba tries to replace attention with selective memory.