Linear Attention Is All You Need. Self-attention at a fraction of the… | by Sam Maddrell-Mander | Towards Data Science

This is the kind of thing anyone who’s spent much time working with transformers and self-attention will have heard a hundred times. It’s both absolutely true, we’ve all experienced this as you try to increase the context size of your model everything suddenly comes to a grinding halt. But then at the same time, virtually every week it seems, there’s a new state of the art model with a new record breaking context length. (Gemini has context length of 2M tokens!) There are lots of sophisticated methods like RingAttention that make training incredibly long context lengths in large distributed systems possible, but what I’m interested in today is a simpler question. How far can we get with linear attention alone? This will be a bit of a whistle stop tour, but bear with me as we touch on a few key points before digging into the results. We can basically summarise the traditional attention mechanism with two key points: This is expressed in the traditional form as: It turns out if we ask ou

Linear Attention Is All You Need | Towards Data Science Skip to content Large Language Models Linear Attention Is All You Need Self-attention at a fraction of the cost? Sam Maddrell-Mander Jun 2, 2024 10 min read Share Photo by Guillaume Jaillet on Unsplash "Attention scales badly with long sequence lengths" This is the kind of thing anyone who’s spent much time working with transformers and self-attention will have heard a hundred times. It’s both absolutely true, we’ve all experienced this as you try to increase the context size of your model everything suddenly comes to a

related reading

• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• transformer_attention.pdfarxiv.org
• 1706.03762arxiv.org
• Overleaf Examplearxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Annotated Transformernlp.seas.harvard.edu
• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org