A Introduction to Transformers

The transformer is a neural network component that can be used to learn useful representations of sequences or sets of datapoints. The transformer has driven recent advances in natural language processing, computer vision, and spatio-temporal modelling. There are many introductions to transformers, but most do not contain precise mathematical descriptions of the architecture and the intuitions behind the design choices are often also missing. Moreover, as research takes a winding path, the explanations for the components of the transformer can be idiosyncratic. In this note we aim for a mathematically precise, intuitive, and clean description of the transformer architecture.

An Introduction to Transformers Richard E. Turner Department of Engineering, University of Cambridge, UK Microsoft Research, Cambridge, UK ret26@cam.ac.uk Abstract. The transformer is a neural network component that can be used to learn useful represen- tations of sequences or sets of data-points [Vaswani et al., 2017]. The transformer has driven…

saved by

• Kalie C

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Everything About Transformerskrupadave.com
• transformer_attention.pdfarxiv.org
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• 9 Transformers – 6.390 - Intro to Machine Learningintroml.mit.edu
• Thinking like Transformersrush.github.io
• Transformers Explained Visually (Part 1): Overview of Functionality | Towards Data Sciencetowardsdatascience.com
• The Annotated Transformernlp.seas.harvard.edu