✳flâneur — a map of the web's best reading

How To Scale Your Model

jax-ml.github.io · 2,176 words · saved by 26 readers

Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered

How To Scale Your Model How to Scale Your Model A Systems View of LLMs on TPUs (Part 0: Intro | Part 1: Rooflines ) Training LLMs often feels like alchemy, but understanding and optimizing the performance of your models doesn't have to. This book aims to demystify the science of scaling language models: how TPUs (and GPUs) work and how they communicate with each other, how LLMs run on real hardware, and how to parallelize your models during training and inference so they run efficiently at massive scale. If you've ever wondered "how expensive should this LLM be to train" or "how much memory do

Explore this link on the map →

saved by

related reading