How To Scale Your Model
Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered
How To Scale Your Model How to Scale Your Model A Systems View of LLMs on TPUs (Part 0: Intro | Part 1: Rooflines ) Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered "how expensive should this LLM be to train" or "how much memory do
saved by
- Tazik Sh
- Karan MJ
- Tasha Pais
- Claire Wang
- Ratan Kaliani
- Ayodeji Adesegun
- Nikki Guo
- Rishi Kothari
- Benedict Neo
- Asher P
- Sarah Pan
- Jianmin Chen
related reading
- Making Deep Learning go Brrrr From First Principleshorace.io
- Scaling is subtler than it seemsberen.io
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- The Scaling Hypothesis · Gwern.netgwern.net
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Demystify Transformers: A Guide to Scaling Laws | by Yu-Cheng Tsai | Sage Ai | Mediummedium.com