All the Transformer Math You Need to Know | How To Scale Your Model

Here we'll do a quick review of the Transformer architecture, specifically how to calculate FLOPs, bytes, and other quantities of interest.

All the Transformer Math You Need to Know | How To Scale Your Model All the Transformer Math You Need to Know Part 4 of How To Scale Your Model ( Part 3: Sharding | Part 5: Training ) Here we'll do a quick review of the Transformer architecture, specifically how to calculate FLOPs, bytes, and other quantities of interest. Authors Affiliation Jacob Austin Google DeepMind Sholto Douglas Roy Frostig Anselm Levskaya Charlie Chen Sharad Vikram Federico Lebron Peter Choy Vinay Ramasesh Albert Webson Reiner Pope * Published Feb. 4, 2025 Counting Dots Let’s start with vectors \(x\), \(y\) and matrices

saved by

• Yudhister Joel Kumar
• Ali Ahmad

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• How To Scale Your Modeljax-ml.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Transformers from Scratche2eml.school
• Inside the Transformer: The Life of a Token - Aleksa Gordićaleksagordic.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• The Annotated Transformernlp.seas.harvard.edu
• transformer_attention.pdfarxiv.org