✳flâneur — a map of the web's best reading
Transformer Math 101 | EleutherAI Blog
blog.eleuther.ai · 2,554 words · saved by 6 readers
We present basic math related to computation and memory usage for transformers
Introduction # A lot of basic, important information about transformer language models can be computed quite simply. Unfortunately, the equations for this are not widely known in the NLP community. The purpose of this document is to collect these equations along with related knowledge about where they come from and why they matter. Note: This post is primarily concerned with training costs, which are dominated by VRAM considerations. For an analogous discussion of inference costs with a focus on latency, check out this excellent blog post by Kipply. Compute Requirements # The basic equation gi
Explore this link on the map →saved by
related reading
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How To Scale Your Modeljax-ml.github.io
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Making Deep Learning go Brrrr From First Principleshorace.io
- Overleaf Examplearxiv.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- All the Transformer Math You Need to Know | How To Scale Your Modeljax-ml.github.io
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
- The Little Book of Deep Learningfleuret.org
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com