[2207.09238] Formal Algorithms for Transformers

Abstract:This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

19 July 2022 Formal Algorithms for Transformers Mary Phuong1 and Marcus Hutter1 1 DeepMind This document aims to be a self-contained, mathematically precise overview of transformer architec- tures and algorithms (not results). It covers what transformers are, how they are trained, what they are used for, their key architectural components,…

related reading

• Formal Algorithms for Transformersarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• transformer_attention.pdfarxiv.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformers from Scratche2eml.school
• A Introduction to Transformersarxiv.org
• 1706.03762arxiv.org
• Transformers Laid Outgoyalpramod.github.io
• A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
• Transformers from Scratchbrandonrohrer.com
• Everything About Transformerskrupadave.com