[2207.09238] Formal Algorithms for Transformers

This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs.

View PDF HTML (experimental) Abstract:This document aims to be a self-contained, mathematically precise overview of transformer architectures and algorithms (*not* results). It covers what transformers are, how they are trained, what they are used for, their key architectural components, and a preview of the most prominent models. The reader is assumed to be familiar with basic ML terminology and simpler neural network architectures such as MLPs. Comments: 16 pages, 15 algorithms Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Neural…

saved by

• Molly Xu
• Andrew Chen

related reading

• [2207.09238] Formal Algorithms for Transformersarxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• A Introduction to Transformersarxiv.org
• Thinking like Transformersrush.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• transformer_attention.pdfarxiv.org
• Transformers from Scratche2eml.school
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Transformer (deep learning) - Wikipediaen.wikipedia.org