flâneur

[2507.04239] Scaling Context Requires Rethinking Attention

arxiv.org · 7,961 words · saved by 1 readers

Abstract:We argue that neither transformers nor sub-quadratic architectures are well suited to training at long sequence lengths: the cost of processing the context is too expensive in the former, too inexpensive in the latter. Approaches such as sliding window attention which reduce the cost-per-token of a transformer impair in-context learning, and so are also unsuitable. To address these limitations, we introduce power attention, an architectural layer for linear-cost sequence modeling whose state size can be adjusted independently of parameters, unlocking the advantages of linear attention on practical domains. We develop and open-source a set of GPU kernels for efficient power attention, identifying a novel pattern of operation fusion to avoid memory and bandwidth bottlenecks. Our experiments on the in-context learning of power attention shows that these models dominate both exponential attention and linear attention at long-context training.

Scaling Context Requires Rethinking Attention Carles Gelada∗ Jacob Buckman∗ Sean Zhang∗ Txus Bach Manifest AI Manifest AI Manifest AI Manifest AI Abstract arXiv:2507.04239v1 [cs.LG] 6 Jul 2025 We argue that neither transformers nor sub-quadratic architectures are well suited…

saved by

related reading