Stand-Alone Self-Attention in Vision Models - NeurIPS-2019-stand-alone-self-attention-in-vision-models-Paper.pdf

N/A

Stand-Alone Self-Attention in Vision Models Prajit Ramachandran∗ Niki Parmar∗ Ashish Vaswani∗ Irwan Bello Anselm Levskaya† Jonathon Shlens Google Research, Brain Team {prajit, nikip, avaswani}@google.com Abstract Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in…

saved by

• Linus Leong

related reading

• transformer_attention.pdfarxiv.org
• [2203.09795] Three things everyone should know about Vision Transformersarxiv.org
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 1706.03762arxiv.org
• Yann LeCun (@ylecun) on Xx.com
• [2201.03545] A ConvNet for the 2020sarxiv.org
• 11.7. The Transformer Architecture — Dive into Deep Learning 1.0.3 documentationd2l.ai
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Hyena Hierarchy: Towards Larger Convolutional Language Models · Hazy Researchhazyresearch.stanford.edu
• [2607.07953] Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routingarxiv.org
• What is an attention mechanism? | IBMibm.com