5D parallelism in LLM training - gdymind's Blog

source: The Ultra-scale Playbook 0. High-level overview Targed on large-scale (like 512 GPUs) training Tradeoff among the following factors memory usage: params, optimizer states, gradients compute ef

5D parallelism in LLM training source: The Ultra-scale Playbook 0. High-level overview Targed on large-scale (like 512 GPUs) training Tradeoff among the following factors memory usage : params, optimizer states, gradients compute efficiency be efficient on a single GPU. e.g., no uncessary transposes ensure the efficiency scales for more GPUs communication overhead Cheatsheet: https://nanotron-ultrascale-playbook.static.hf.space/assets/images/ultra-cheatsheet.svg will expalin the details below, here is just a summary when to use which parallelism memory reduction: see which part of the memory i

saved by

• Jeff Brown

related reading

• How To Scale Your Modeljax-ml.github.io
• Paradigms of Parallelism | Colossal-AIcolossalai.org
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Visualizing 6D Mesh Parallelism · mainmain-horse.github.io
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Pretraining parallelisms and failed runs — Flashcardsflashcards.dwarkesh.com
• Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
• Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com