✳flâneur — a map of the web's best reading
Visualizing 6D Mesh Parallelism · main
main-horse.github.io · 3,660 words · saved by 1 readers
Plus some lore
Table of Contents Table of Contents Last updated 23 Dec 2024 This is a companion longpost for a fun project I may leave unpublished. In here, I show the reader how I personally visualize the collective communications involved in a simple 2⁶ 6D parallel mesh: There are many articles that describe various training parallelisms in vague words and simple visuals. Most of them fail to convey a deep understanding of the exact communications involved in a single training step, and even for the outliers that do, they do not cover the more complex case of combining all approaches . So, in this post, I
Explore this link on the map →related reading
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Pretraining parallelisms and failed runs — Flashcardsflashcards.dwarkesh.com
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- How To Scale Your Modeljax-ml.github.io
- 👨👩👧👦 Distributed Training - Composerdocs.mosaicml.com
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly