Transformer Math 101 | EleutherAI Blog

We present basic math related to computation and memory usage for transformers

Introduction # A lot of basic, important information about transformer language models can be computed quite simply. Unfortunately, the equations for this are not widely known in the NLP community. The purpose of this document is to collect these equations along with related knowledge about where they come from and why they matter. Note: This post is primarily concerned with training costs, which are dominated by VRAM considerations. For an analogous discussion of inference costs with a focus on latency, check out this excellent blog post by Kipply. Compute Requirements # The basic equation gi

saved by

• Claire Wang
• Shubham Chandel
• Atem Aguer
• Andrew Chen
• James Hennessy
• Vincent Huang

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• How To Scale Your Modeljax-ml.github.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• All About Transformer Inferencejax-ml.github.io
• All About Rooflines | How To Scale Your Modeljax-ml.github.io
• Estimating training compute of deep learning models | Epoch AIepochai.org
• 5D parallelism in LLM training - gdymind's Bloggdymind.com
• Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com