Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

N/A

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos 1 2 Apoorv Vyas 1 2 Nikolaos Pappas 3 François Fleuret 2 4 * Abstract by the global receptive field of self-attention, which pro- arXiv:2006.16236v3 [cs.LG] 31 Aug 2020 Transformers achieve remarkable performance in…

related reading

• transformer_attention.pdfarxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• The Annotated Transformernlp.seas.harvard.edu
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• Log-Linear Attentionarxiv.org
• 1706.03762arxiv.org
• [2011.04006] Long Range Arena: A Benchmark for Efficient Transformersarxiv.org
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
• Attention is All you Needproceedings.neurips.cc
• [1807.03819] Universal Transformersarxiv.org
• Parallelizing Linear Transformers with the Delta Rule over Sequence Lengtharxiv.org
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org