Visualizing 6D Mesh Parallelism · main

Plus some lore

Table of Contents Table of Contents Last updated 23 Dec 2024 This is a companion longpost for a fun project I may leave unpublished. In here, I show the reader how I personally visualize the collective communications involved in a simple 2⁶ 6D parallel mesh: There are many articles that describe various training parallelisms in vague words and simple visuals. Most of them fail to convey a deep understanding of the exact communications involved in a single training step, and even for the outliers that do, they do not cover the more complex case of combining all approaches . So, in this post, I

related reading

• 5D parallelism in LLM training - gdymind's Bloggdymind.com
• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Paradigms of Parallelism | Colossal-AIcolossalai.org
• ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
• The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
• irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
• Pretraining parallelisms and failed runs — Flashcardsflashcards.dwarkesh.com
• Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
• Parallelism methods · Hugging Facehuggingface.co
• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com