Parallelizing Linear Transformers with the Delta Rule over Sequence Length

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Parallelizing Linear Transformers with the Delta Rule over Sequence Length Songlin Yang ⋄ Bailin Wang ⋄ Yu Zhang † Yikang Shen ‡ Yoon Kim ⋄ ⋄ Massachusetts Institute of Technology † Soochow University ‡ MIT-IBM Watson AI Lab yangsl66@mit.edu Abstract Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transfor

saved by

• Clementine Yang

related reading

• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• DeltaNet Explained (Part II) | Songlin Yangsustcsonglin.github.io
• transformer_attention.pdfarxiv.org
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• [2412.06464] Gated Delta Networks: Improving Mamba2 with Delta Rulearxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• Log-Linear Attentionarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• [2607.07953] Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingarxiv.org
• Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org