Transformer Math 101 | EleutherAI Blog

We present basic math related to computation and memory usage for transformers

Introduction # A lot of basic, important information about transformer language models can be computed quite simply. Unfortunately, the equations for this are not widely known in the NLP community. The purpose of this document is to collect these equations along with related knowledge about where they come from and why they matter. Note: This post is primarily concerned with training costs, which are dominated by VRAM considerations. For an analogous discussion of inference costs with a focus on latency, check out this excellent blog post by Kipply. Compute Requirements # The basic equation gi

saved by

• Claire Wang
• Shubham Chandel
• Atem Aguer
• Andrew Chen
• James Hennessy
• Vincent Huang

related reading

• Transformer Inference Arithmetic | kipply's blogkipp.ly
• How To Scale Your Modeljax-ml.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
• Overleaf Examplearxiv.org
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Reducing Activation Recomputationarxiv.org
• 1910.02054v3arxiv.org
• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
• 5D parallelism in LLM training - gdymind's Bloggdymind.com