An Intuition for Attention | Jay Mody

Deriving the equation for scaled dot product attention.

An Intuition for Attention | Jay Mody An Intuition for Attention October 22, 2022 ChatGPT and other large language models use a special type of neural network called the transformer. The transformer defining feature is the attention mechanism. Attention is defined by the equation: \[\text{attention}(Q, K, V) = \text{softmax}(\frac{QK^T}{\sqrt{d_k}})V\] Attention can come in different forms, but this version of attention (known as scaled dot product attention) was first proposed in the original transformer paper . In this post, we'll build an intuition for the above equation by deriving it from

saved by

• Saeejith Nair
• Nima Pourjafar

related reading

• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• Understanding Attention in LLMs | Bartosz Milewski's Programming Cafebartoszmilewski.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• neural networks - What exactly are keys, queries, and values in attention mechanisms? - Cross Validatedstats.stackexchange.com
• Some Intuition on Attention and the Transformereugeneyan.com
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Attention Is Off By One – Evan Millerevanmiller.org
• The math behind Attention: Keys, Queries, and Values matricesyoutube.com