Linear Attention Fundamentals | Hailey Schoelkopf

The basics of linear attention in sub-quadratic language model architectures.

Linear Attention Fundamentals | Hailey Schoelkopf Linear Attention Fundamentals The basics of linear attention in sub-quadratic language model architectures. Introduction This post will be a short overview recapping key formulas and intuitions around the increasingly-popular family of methods under the umbrella of Linear Attention, first introduced by Katharopoulos et al. (2020) . The material in this post is also covered excellently by Yang, Wang et al. (2023) and Yang et al. (2024). This post will assume familiarity with the transformer architecture and softmax attention, and with KV caching

saved by

• Sarah Pan
• Jirat C
• Nathan Chen
• Arya G
• Prem Kumar

related reading

• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• Biao's Bloghebiao064.github.io
• Test-Time Training with KV Binding Is Secretly Linear Attentionresearch.nvidia.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• Log-Linear Attentionarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• How Attention Got So Efficient [GQA/MLA/DSA]youtube.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• transformer_attention.pdfarxiv.org
• Parallelizing Linear Transformers with the Delta Rule over Sequence Lengtharxiv.org
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentionarxiv.org