[2207.09238] Formal Algorithms for Transformers

Abstract:This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

19 July 2022 Formal Algorithms for Transformers Mary Phuong1 and Marcus Hutter1 1 DeepMind This document aims to be a self-contained, mathematically precise overview of transformer architec- tures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components,…

related reading

• Formal Algorithms for Transformersarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• transformer_attention.pdfarxiv.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• A Introduction to Transformersarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Thinking like Transformersrush.github.io
• Everything About Transformerskrupadave.com
• Transformers Laid Outgoyalpramod.github.io