All the Transformer Math You Need to Know | How To Scale Your Model

Here we'll do a quick review of the Transformer architecture, specifically how to calculate FLOPs, bytes, and other quantities of interest.

All the Transformer Math You Need to Know | How To Scale Your Model All the Transformer Math You Need to Know Part 4 of How To Scale Your Model ( Part 3: Sharding | Part 5: Training ) Here we'll do a quick review of the Transformer architecture, specifically how to calculate FLOPs, bytes, and other quantities of interest. Authors Affiliation Jacob Austin Google DeepMind Sholto Douglas Roy Frostig Anselm Levskaya Charlie Chen Sharad Vikram Federico Lebron Peter Choy Vinay Ramasesh Albert Webson Reiner Pope * Published Feb. 4, 2025 Counting Dots Let’s start with vectors \(x\), \(y\) and matrices

saved by

• Yudhister Joel Kumar
• Ali Ahmad
• nathan karri

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• How To Scale Your Modeljax-ml.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
• The Annotated Transformernlp.seas.harvard.edu
• All About Transformer Inferencejax-ml.github.io
• How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io