# Hybrid attention–SSM architectures
A hybrid attention–SSM architecture is a language-model stack that interleaves a minority of full softmax-attention layers with a majority of linear-time layers, state-space model (SSM) layers, or linear-attention recurrent layers, so that the model keeps exact content lookup from attention while most of the sequence processing runs through fixed-size recurrent states. The design appeared as a research direction in the early 2020s and reached production scale in early 2024, with [Google DeepMind](https://www.edgepedia.org/google-deepmind)'s Griffin in February 2024, [AI21 Labs](https://www.edgepedia.org/ai21-labs)' Jamba in March 2024, and later MiniMax's 400-billion-parameter model, Qwen3-Next, and [Moonshot AI](https://www.edgepedia.org/moonshot-ai)'s Kimi Linear in October 2025.
| System | Maker and date | Design | Ratio (linear:full) | Scale and context |
|---|---|---|---|---|
| Jamba | AI21 Labs, March 2024 | Mamba SSM + attention + MoE | 7:1 (1 attention per 8 layers) | 12B active / 52B total; 256K context[1](https://arxiv.org/html/2403.19887v2) |
| Griffin / Hawk | Google DeepMind, February 2024 | RG-LRU gated recurrences + local attention | local attention mixed with recurrences | up to 14B parameters[2](https://arxiv.org/html/2402.19427) |
| MiniMax | MiniMax, 2025 | lightning attention hybrid | 6–7:1 | 400B parameters[3](https://arxiv.org/html/2507.06457) |
| Qwen3-Next | Qwen Team, 2025 | gated DeltaNet family + full attention | 3:1 (1 full-attention layer per 4) | production long-context model[4](https://d2l.smola.org/chapter_recurrent-modern/hybrids.html) |
| Kimi Linear | Moonshot AI, October 2025 | Kimi Delta Attention + Multi-Head Latent Attention | 3:1 | 3B activated / 48B total; 1M context[5](https://arxiv.org/pdf/2510.26692) |
| RecurrentGemma | Google, 2024 | Griffin design with sliding-window attention | 2:1 | 2B and 9B parameters[3](https://arxiv.org/html/2507.06457) |
| Mamba-2-Hybrid | NVIDIA, 2024 | Mamba-2 + attention | 4:1 | 8B parameters[3](https://arxiv.org/html/2507.06457) |
## What a hybrid architecture is
The two layer types do complementary jobs. Softmax attention computes a score between every pair of tokens, so its computation grows quadratically with sequence length, and during autoregressive decoding each attention layer keeps a key-value (KV) cache that grows with the context. Recurrent layers such as Gated DeltaNet or Mamba-2 instead carry a fixed-size state that is updated once per token, so cost per token is constant and the state does not grow[6](https://sebastianraschka.com/llm-architecture-gallery/hybrid-attention/). The recurrent layers are cheap and compact but compress history lossily; the attention layers retain direct access to every past token.
A hybrid stack therefore uses most layers for recurrent or linear-attention processing and reserves a smaller number of softmax-attention layers for direct content lookup[6](https://sebastianraschka.com/llm-architecture-gallery/hybrid-attention/). The division of labor is measurable: ablating the few full-attention layers of a deployed hybrid reduces needle-in-a-haystack retrieval to roughly zero, showing that the attention layers carry the exact-retrieval load[4](https://d2l.smola.org/chapter_recurrent-modern/hybrids.html).
## How the mechanism works
Recall, not perplexity, is what the ratio controls. A July 2025 systematic study trained 72 models, 36 at 340M parameters on 20B tokens, and 36 at 1.3B parameters on 100B tokens, covering six linear-attention variants across five hybridization ratios[3](https://arxiv.org/html/2507.06457). It found that the linear-to-full ratio primarily controls recall, while language-modeling loss is comparatively insensitive to it: moving from a 24:1 to a 3:1 ratio nearly doubles RULER (benchmark measuring long-context retrieval ability) recall while shifting language-model loss by less than 1[3](https://arxiv.org/html/2507.06457). Pure linear configurations score around 0.1–0.35 on RULER against a full-attention baseline of about 0.42, and most architectures approach or exceed that baseline at a 3:1 ratio, with DeltaNet and Gated-DeltaNet peaking there[3](https://arxiv.org/html/2507.06457). Long-context quality rises steeply once a few full-attention blocks or heads are present, after which perplexity plateaus[3](https://arxiv.org/html/2507.06457).
This explains why sparse ratios work. Because loss barely moves as attention layers are removed, a model can look nearly identical on standard benchmarks while differing sharply on retrieval; a handful of attention layers restore most of the recall that pure recurrence loses.
## Origin and lineage
Jamba combined [Transformer](https://www.edgepedia.org/transformer) layers with [Mamba](https://www.edgepedia.org/mamba-deep-learning-architecture), a recent state-space model, plus a mixture-of-experts module, in March 2024[1](https://arxiv.org/html/2403.19887v2). Griffin, from Google DeepMind in February 2024, took a different route: Hawk is an RNN with gated linear recurrences, and Griffin mixes those recurrences with local attention[2](https://arxiv.org/html/2402.19427). A systematic survey catalogs the wider design space that grew alongside them: Hybrid-H3, StripedHyena (1:1, 7B), RecurrentGemma, Mamba-2-Hybrid, Zamba, Samba, YOCO, and Hymba[3](https://arxiv.org/html/2507.06457). The sources do not settle who first proposed combining the two layer types or how credit divides among the S4/Mamba, DeltaNet, and linear-attention lines; that attribution question remains open in the retrieved evidence.
## Named systems and their designs
**Jamba** interleaves Mamba and attention layers at a 1:7 attention-to-Mamba ratio, chosen through ablations as the most compute-efficient variant among the best-performing ones in quality, with the MoE module applied every 2 layers[1](https://arxiv.org/html/2403.19887v2). The released model has 12B active and 52B total parameters, supports a 256K-token context, described at release as the longest for production-grade publicly available models, and fits on a single 80GB GPU with 8-bit weights even at contexts over 128K tokens[1](https://arxiv.org/html/2403.19887v2).
**Griffin** mixes gated linear recurrences (RG-LRU) with local attention and scales to 14B parameters; its companion pure-RNN model Hawk exceeds the reported performance of Mamba on downstream tasks, and Griffin matches Llama-2 despite being trained on over 6 times fewer tokens[2](https://arxiv.org/html/2402.19427).
**Kimi Linear** uses Kimi Delta Attention (KDA), an extension of Gated DeltaNet with finer-grained gating, interleaved with Multi-Head Latent Attention in a uniform 3:1 ratio, pretrained at 3B activated and 48B total parameters[5](https://arxiv.org/pdf/2510.26692). Its ratio ablation over {0:1, 1:1, 3:1, 7:1} found 3:1 gave the lowest training and validation losses; 7:1 had significantly worse validation performance, 1:1 added inference overhead, and the pure full-attention baseline performed poorly[5](https://arxiv.org/pdf/2510.26692).
**Qwen3-Next** fixes one full-attention layer per four layers using the gated-DeltaNet family[4](https://d2l.smola.org/chapter_recurrent-modern/hybrids.html). **[MiniMax](https://www.edgepedia.org/minimax-model-family)** converged on a 6–7:1 ratio and scaled the approach to 400 billion parameters while maintaining competitive common reasoning performance, according to the survey citing MiniMax's team[3](https://arxiv.org/html/2507.06457); no primary MiniMax source was retrieved for this article, so its lightning-attention design details are known here second-hand.
## By the numbers: efficiency and quality
Vendor-reported figures set the headline claims. AI21 reported that Jamba's long-context throughput is 3x that of Mixtral-8x7B, and that its [KV cache](https://www.edgepedia.org/kv-cache) at 256K context in 16-bit is 4GB, versus 32GB for Mixtral, and 128GB for Llama-2 70B, an 8x smaller cache than a vanilla Transformer[1](https://arxiv.org/html/2403.19887v2). Moonshot reported that Kimi Linear reduces KV cache usage by up to 75% and achieves up to 6x decoding throughput at a 1M context versus full MLA[5](https://arxiv.org/pdf/2510.26692). DeepMind reported that Hawk and Griffin achieve significantly higher throughput than MQA Transformers and lower latency when sampling long sequences[2](https://arxiv.org/html/2402.19427).
Independent measurements are consistent on the memory side. The 72-model study found that a gated hierarchical backbone such as HGRN-2 or GatedDeltaNet with one softmax-attention layer per 3–6 linear layers achieves near-Transformer recall while shrinking the KV cache by a factor of 4–7 at 1.3B parameters[3](https://arxiv.org/html/2507.06457). On retrieval, AI21 reported that Jamba shows excellent needle-in-a-haystack performance given that its implementation uses only 4 attention layers[1](https://arxiv.org/html/2403.19887v2).
## Limits and disputes
**Recall deficits of pure recurrence are well documented.** In NVIDIA's controlled 8B-parameter comparison (Waleffe et al. 2024), a pure Mamba-2 model matched a transformer on perplexity and most benchmarks while scoring 29% on five-shot MMLU against the transformer's 46%[4](https://d2l.smola.org/chapter_recurrent-modern/hybrids.html). Its phone-book recall collapsed once the book grew past a few hundred tokens, a failure mode the authors call fuzzy memory: the model returns an answer that shares digits with the right one[4](https://d2l.smola.org/chapter_recurrent-modern/hybrids.html).
**Copying and exact retrieval lag without attention or fine-tuning.** DeepMind reported that pre-trained Hawk and Griffin perform less well than [Transformers](https://www.edgepedia.org/transformers) on copying and exact-retrieval tasks without fine-tuning, though Griffin performs better than Transformers on sequences longer than those seen during training[2](https://arxiv.org/html/2402.19427).
**Do hybrids match full attention?** The vendors and the independent literature disagree. Moonshot claims Kimi Linear is the first hybrid linear architecture to outperform full attention under fair comparisons across various scenarios[5](https://arxiv.org/pdf/2510.26692). Independent analyses find hybrids approach, not exceed, the full-attention recall baseline, and that pure linear models show large deficits (0.1–0.35 versus about 0.42 RULER)[3](https://arxiv.org/html/2507.06457). The optimal ratio is likewise unresolved: Jamba's ablations favored 1:7 attention-to-Mamba as most compute-efficient among top-quality variants[1](https://arxiv.org/html/2403.19887v2), while the independent study finds recall demands a denser 3:1-to-6:1 linear-to-full ratio, with most architectures only reaching full-attention recall at 3:1[3](https://arxiv.org/html/2507.06457). Part of the difference is scope: the independent work measures recall directly, whereas compute-efficiency ablations weigh quality more broadly.
## References
1. Jamba: A Hybrid Transformer-Mamba Language Model (AI21 Labs) — https://arxiv.org/html/2403.19887v2
2. Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (Google DeepMind) — https://arxiv.org/html/2402.19427
3. A Systematic Analysis of Hybrid Linear Attention — https://arxiv.org/html/2507.06457
4. 12.7 Hybrid Architectures, Dive into Deep Learning — https://d2l.smola.org/chapter_recurrent-modern/hybrids.html
5. Kimi Linear: An Expressive, Efficient Attention Architecture (Moonshot AI) — https://arxiv.org/pdf/2510.26692
6. Hybrid Attention, LLM Architecture Gallery (Sebastian Raschka) — https://sebastianraschka.com/llm-architecture-gallery/hybrid-attention/
7. Understanding and Enhancing Mamba-Transformer Hybrids for Memory Recall and Language Modeling (ACL BabyLM 2025) — https://aclanthology.org/2025.babylm-main.27.pdf
---
*Topic: Encyclopedia › Technology and the built world › Computing and digital systems › Modern AI: foundation models, generative AI, and the AI industry › Foundation-model methods and training › Large language model architecture and scaling*
*Initially written Sep 17, 2026 · Reviewed: Sep 30, 2026 · Edited: Sep 30, 2026; Oct 11, 2026 · Last review: Sep 30, 2026*
*Copyright 2026 EdgeChat AI, a subsidiary of Biostate AI.*
License: Edgepedia Community License 1.0, https://www.edgepedia.org/edgepedia/license