Training Models Too Big for One GPU | ZeRO & FSDP - YouTube

0 of 8 lessons complete

How do you train a model that does not fit on one GPU? In this video, we cover how GPUs represent and store real numbers, mixed-precision training, and the Zero Redundancy Optimizer (ZeRO) and FSDP to reduce memory redundancy in data parallelism. Chapters 0:00 How a model learns 0:55 Adam optimizer 1:56 How a computer stores real numbers 11:40 The memory constraint 12:40 Data parallelism 13:21 How GPUs sync their gradients 16:07 ZeRO stage 1 and 2: shard the optimizer states, then the gradients 19:27 ZeRO stage 3: shard the weights 21:28 FSDP: the same schedule one unit at a time…

saved by

• Ishaan Panigrahi

related reading

• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Your Softmax Is Broken. Here's the Fix.youtube.com
• How trillion-parameter AI models work and how to serve themyoutube.com
• How GPT, Claude, and Gemini are actually trained and served – Reiner Popeyoutube.com
• Quantization from the ground upngrok.com
• 1910.02054v3arxiv.org
• Making Deep Learning go Brrrr From First Principleshorace.io
• 5D parallelism in LLM training - gdymind's Bloggdymind.com
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com
• How To Scale Your Modeljax-ml.github.io
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com