Mamba, a State Space Model developed by Gu and Dao, runs up to 5x faster than Transformers and scales linearly with sequence length, handling contexts up to 1 million tokens. The Mamba-3B model outperforms same-size Transformers on language modeling benchmarks and matches Transformers twice its size, both in pretraining and downstream evaluation. It eliminates the quadratic bottleneck of attention, the core architectural constraint that makes long-context Transformers slow and memory-hungry.
Transformers process every token against every previous token, producing O(n squared) training complexity and an O(n) KV cache that grows until CUDA runs out of memory. Mitigations like Sliding Window Attention and FlashAttention exist, but they are marginal fixes. Mamba replaces the attention mechanism entirely with a Control Theory-inspired SSM for inter-token communication, while keeping MLP-style projections for per-token computation.
The full article is worth reading for three specific reasons: it builds the SSM intuition from first principles using a concrete control-theory framing, it traces the architectural lineage from S4 through to the selective state space mechanism that makes Mamba actually work, and it confronts the interpretability and AI safety implications of moving away from attention, the one component researchers understand best. If Transformer dominance is not permanent, this is where the argument starts.
[READ ORIGINAL →]