👨👩👧👦 Distributed Training - Composer
Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism: Composer distributes work across d...
👨👩👧👦 Distributed Training # Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism # Composer distributes work across devices via data-parallelism-only . We made this design choice in order to provide the most flexibility to algorithms, which can modify the training loop in complex ways. Data parallelism greatly simplifies model building and memory management. Every GPU is performing the same work, so inspecting the rank zero is sufficient to reason about memory, performance, and o
saved by
related reading
- ⏯️ Autoresume Training - Composerdocs.mosaicml.com
- ⏱️ Performance Profiling - Composerdocs.mosaicml.com
- ⚡ Migrating from PTL - Composerdocs.mosaicml.com
- ♻️ Auto Microbatching - Composerdocs.mosaicml.com
- 🥽 Analyzing Tracesdocs.mosaicml.com
- Accelerated Inference for Large Transformer Models Using NVIDIA Triton Inference Server | NVIDIA Technical Blogdeveloper.nvidia.com
- Composer2.pdfcursor.com
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai