Parallelizing Linear Transformers with the Delta Rule over Sequence Length

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Parallelizing Linear Transformers with the Delta Rule over Sequence Length Songlin Yang ⋄ Bailin Wang ⋄ Yu Zhang † Yikang Shen ‡ Yoon Kim ⋄ ⋄ Massachusetts Institute of Technology † Soochow University ‡ MIT-IBM Watson AI Lab yangsl66@mit.edu Abstract Transformers with linear attention (i.e., linear transformers) and state-space models have recently been suggested as a viable linear-time alternative to transformers with softmax attention. However, these models still underperform transformers especially on tasks that require in-context retrieval. While more expressive variants of linear transfor

saved by

• Clementine Yang

related reading

• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• DeltaNet Explained (Part II) | Songlin Yangsustcsonglin.github.io
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Overleaf Examplearxiv.org
• The Annotated Transformernlp.seas.harvard.edu