How To Scale Your Model
Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered
How To Scale Your Model How to Scale Your Model A Systems View of LLMs on TPUs (Part 0: Intro | Part 1: Rooflines ) Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered "how expensive should this LLM be to train" or "how much memory do
saved by
- Tazik Sh
- Karan MJ
- Tasha Pais
- Claire Wang
- Ratan Kaliani
- Ayodeji Adesegun
- Nikki Guo
- Rishi Kothari
- Benedict Neo
- Asher P
- Sarah Pan
- Jianmin Chen
related reading
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- TPU Deep Divehenryhmko.github.io
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com
- Chinchillaarxiv.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Bits, FLOPS, and Watts: A Systems-Level Perspective of Scaling LLMs — Part 1 | by Asheesh Goja | Mediummedium.com