✳flâneur — a map of the web's best reading

👨‍👩‍👧‍👦 Distributed Training - Composer

docs.mosaicml.com · 3,956 words · saved by 1 readers

Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism: Composer distributes work across d...

👨‍👩‍👧‍👦 Distributed Training # Composer supports distributed training on multiple devices, whether it be multiple GPUs on a single node or multiple GPUs across multiple nodes. Data Parallelism # Composer distributes work across devices via data-parallelism-only . We made this design choice in order to provide the most flexibility to algorithms, which can modify the training loop in complex ways. Data parallelism greatly simplifies model building and memory management. Every GPU is performing the same work, so inspecting the rank zero is sufficient to reason about memory, performance, and o

Explore this link on the map →

saved by

related reading