5D parallelism in LLM training - gdymind's Blog

source: The Ultra-scale Playbook 0. High-level overview Targed on large-scale (like 512 GPUs) training Tradeoff among the following factors memory usage: params, optimizer states, gradients compute ef

5D parallelism in LLM training source: The Ultra-scale Playbook 0. High-level overview Targed on large-scale (like 512 GPUs) training Tradeoff among the following factors memory usage : params, optimizer states, gradients compute efficiency be efficient on a single GPU. e.g., no uncessary transposes ensure the efficiency scales for more GPUs communication overhead Cheatsheet: https://nanotron-ultrascale-playbook.static.hf.space/assets/images/ultra-cheatsheet.svg will expalin the details below, here is just a summary when to use which parallelism memory reduction: see which part of the memory i

saved by

• Jeff Brown

related reading

• How To Scale Your Modeljax-ml.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Making Deep Learning go Brrrr From First Principleshorace.io
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Parallelism methods · Hugging Facehuggingface.co
• Breadth-First Pipeline Parallelismarxiv.org
• Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Visualizing 6D Mesh Parallelism · mainmain-horse.github.io