A Introduction to Transformers

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of datapoints. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture.

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of datapoints. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathem

saved by

• Kalie C

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• [2207.09238] Formal Algorithms for Transformersarxiv.org
• Everything About Transformerskrupadave.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer Circuits Threadtransformer-circuits.pub
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transformers Explained Visually (Part 1): Overview of Functionality | Towards Data Sciencetowardsdatascience.com