Linear Attention Fundamentals | Hailey Schoelkopf

The basics of linear attention in sub-quadratic language model architectures.

Linear Attention Fundamentals | Hailey Schoelkopf Linear Attention Fundamentals The basics of linear attention in sub-quadratic language model architectures. Introduction This post will be a short overview recapping key formulas and intuitions around the increasingly-popular family of methods under the umbrella of Linear Attention, first introduced by Katharopoulos et al. (2020) . The material in this post is also covered excellently by Yang, Wang et al. (2023) and Yang et al. (2024). This post will assume familiarity with the transformer architecture and softmax attention, and with KV caching

saved by

• Jirat C
• Nathan Chen

related reading

• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• Log-Linear Attentionarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Parallelizing Linear Transformers with the Delta Rule over Sequence Lengtharxiv.org
• ali (@waterloo_intern) on Xx.com
• [2607.07953] Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingarxiv.org
• Mamba: The Easy Wayjackcook.com
• Linear Attention Is All You Need | Towards Data Sciencetowardsdatascience.com
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Kimi Linear: An Expressive, Efficient Attention Architecturealphaxiv.org