Parallelizing Linear Transformers with the Delta Rule over Sequence Length

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Parallelizing Linear Transformers with the Delta Rule over Sequence Length Songlin Yang ⋄ Bailin Wang ⋄ Yu Zhang † Yikang Shen ‡ Yoon Kim ⋄ ⋄ Massachusetts Institute of Technology † Soochow University ‡ MIT-IBM Watson AI Lab yangsl66@mit.edu Abstract Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transfor

saved by

• Clementine Yang

related reading

• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• DeltaNet Explained (Part II) | Songlin Yangsustcsonglin.github.io
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• [2412.06464] Gated Delta Networks: Improving Mamba2 with Delta Rulearxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• The Annotated Transformernlp.seas.harvard.edu
• Log-Linear Attentionarxiv.org
• Transformers from Scratche2eml.school
• [2607.07953] Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingarxiv.org
• ali (@waterloo_intern) on Xx.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io