[2507.04239] Scaling Context Requires Rethinking Attention
Abstract:We argue that neither transformers nor sub-quadratic architectures are well suited to training at long sequence lengths: the cost of processing the context is too expensive in the former, too inexpensive in the latter. Approaches such as sliding window attention which reduce the cost-per-token of a transformer impair in-context learning, and so are also unsuitable. To address these limitations, we introduce power attention, an architectural layer for linear-cost sequence modeling whose state size can be adjusted independently of parameters, unlocking the advantages of linear attention on practical domains. We develop and open-source a set of GPU kernels for efficient power attention, identifying a novel pattern of operation fusion to avoid memory and bandwidth bottlenecks. Our experiments on the in-context learning of power attention shows that these models dominate both exponential attention and linear attention at long-context training.
Scaling Context Requires Rethinking Attention Carles Gelada∗ Jacob Buckman∗ Sean Zhang∗ Txus Bach Manifest AI Manifest AI Manifest AI Manifest AI Abstract arXiv:2507.04239v1 [cs.LG] 6 Jul 2025 We argue that neither transformers nor sub-quadratic architectures are well suited…
saved by
related reading
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- transformer_attention.pdfarxiv.org
- [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
- 2502.11089arxiv.org
- Linear Transformers Are Faster After All – Manifest AImanifestai.com
- [2509.24552] Short window attention enables long-term memorizationarxiv.org
- [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
- Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
- Log-Linear Attentionarxiv.org
- 2310.01889arxiv.org
- [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
- Extending Context is Hard | kaiokendevkaiokendev.github.io