[2207.09238] Formal Algorithms for Transformers

Abstract:This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• A Introduction to Transformersarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformers from Scratche2eml.school
• transformer_attention.pdfarxiv.org
• Transformer Circuits Threadtransformer-circuits.pub
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• The Transformer Family | Hacker Newsnews.ycombinator.com
• How do Transformers work? · Hugging Facehuggingface.co