ModelRefs / Mamba — AI Glossary

Mamba — AI Glossary

A state space model architecture with selective state expansion that achieves transformer-competitive quality at linear inference cost.

Overview

Gu & Dao (2023) introduced Mamba with a selective scan mechanism that lets the model focus on relevant parts of history without full attention. Mamba models train 5× faster than equivalently-sized transformers at long context. Mamba-2 and hybrid Jamba models combine SSM layers with selective attention.

Reference details

Topic	architecture
Last reviewed	2026-06-24

How it works

The model is a recurrence rather than an attention stack: a hidden state is carried forward and updated at each position, so computation per token does not grow with how much came before.
The decisive addition over earlier recurrent sequence models is input-dependent parameters. The transition that updates the state is a function of the current token, which lets the model decide what to keep and what to discard rather than applying one fixed update.
Making the parameters input-dependent breaks the convolutional trick that made earlier models trainable in parallel, so training uses a hardware-aware parallel scan instead — an associative-scan formulation that computes the recurrence across the sequence without materialising the intermediate states in slow memory.
At inference the state is a fixed-size summary. Generating a token reads and updates that state, so there is no cache that grows with context length.
The same property is the architecture's weakness: everything from the past has been compressed into a constant, so exact recall of a specific earlier detail is harder than for a mechanism that retains every token.
Hybrid designs interleave a few attention layers among the recurrent ones, which restores exact lookup where it is needed while keeping most of the linear-cost benefit.

Example: Where the linear-cost claim actually pays

Attention cost grows with the square of sequence length while a recurrence grows linearly. Going from 1,000 to 100,000 tokens multiplies the sequence by 100, so attention's cost multiplies by 100^2 = 10,000 while the recurrence's multiplies by 100 — a factor of 10,000 / 100 = 100 in relative favour of the recurrence at that length. The memory side is starker because it is what fails first. A 32-layer model of width 4,096 holding keys and values in 16 bits needs 2 x 2 x 32 x 4,096 x 1,000 = 524,288,000 bytes — about 0.52GB — of cache at 1,000 tokens, and 2 x 2 x 32 x 4,096 x 100,000 = 52,428,800,000 bytes, about 52.4GB, at 100,000. The recurrent model's state is the same size at both: that is the entire architectural argument. At 1,000 tokens the 0.52GB is irrelevant and the quadratic term is small, so there is nothing to win; the advantage exists only where sequences are long, which is why these architectures are evaluated on long-context tasks and look unremarkable on short ones.

Commonly confused with

This is not a transformer with a cheaper attention variant. Efficient-attention methods — sparse patterns, low-rank approximations, linearised kernels — keep the attention mechanism and reduce its cost, so every token remains individually addressable. A recurrence replaces attention with a fixed-size state, so past tokens are not addressable at all, only summarised. That is why the failure modes differ in kind rather than degree: an efficient-attention model loses accuracy on the pairs its pattern omits, while a recurrent model loses the ability to retrieve any specific earlier token exactly.

When to use it

Reach for it when:

Very long sequences, where the quadratic term and the growing cache are the binding constraints.
Streaming or continuous input, where a fixed-size state is a natural fit and a growing cache is not.
Memory-constrained deployment at long context, where the cache rather than the weights is what does not fit.
Hybrid designs when exact lookup is also needed, which restore it at a fraction of full attention's cost.

Reach for something else when:

Short contexts, where there is no quadratic cost to avoid and the comparison is decided by other factors.
Tasks requiring exact retrieval of a specific earlier detail, which a compressed fixed state does poorly.
Where the tooling ecosystem matters, since the surrounding infrastructure is far less mature than for attention stacks.
As a drop-in replacement expecting identical behaviour — the retrieval characteristics differ, and that shows up in evaluation rather than in the architecture diagram.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Mamba — AI Glossary.

Frequently asked questions

What is Mamba?