👨👩👧👦 Distributed Training - Composer
Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism: Composer distributes work across d...
👨👩👧👦 Distributed Training # Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism # Composer distributes work across devices via data-parallelism-only . We made this design choice in order to provide the most flexibility to algorithms, which can modify the training loop in complex ways. Data parallelism greatly simplifies model building and memory management. Every GPU is performing the same work, so inspecting the rank zero is sufficient to reason about memory, performance, and o
saved by
related reading
- ⏱️ Performance Profiling - Composerdocs.mosaicml.com
- ♻️ Auto Microbatching - Composerdocs.mosaicml.com
- ⏯️ Autoresume Training - Composerdocs.mosaicml.com
- 🥽 Analyzing Tracesdocs.mosaicml.com
- ⚡ Migrating from PTL - Composerdocs.mosaicml.com
- Composer2.pdfcursor.com
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- State of torch.compile for training (August 2025) : ezyang's blogblog.ezyang.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com