Attention Is Off By One – Evan Miller

Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.

Attention Is Off By One – Evan Miller Attention Is Off By One By Evan Miller July 24, 2023 About which one cannot speak, one must pass over in silence. –Wittgenstein Do you see the off-by-one error in this formula? \[ \textrm{Attention}(Q, K, V) = \textrm{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V \] The attention formula is the central equation of modern AI, but there’s a bug in it that has been driving me nuts the last week. I tried writing a serious-looking research paper about the bug and my proposed fix, but I lost a series of pitched battles against Pytorch and biblatex , so I figured I

saved by

• Ratan Kaliani
• Izu Elechi
• Simon Wu
• Yudhister Joel Kumar
• Aidan Smith
• Andrew Cai
• David Oks

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• The Annotated Transformernlp.seas.harvard.edu
• 1706.03762arxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• Some Intuition on Attention and the Transformereugeneyan.com
• transformer_attention.pdfarxiv.org
• Everything About Transformerskrupadave.com