Efficient Training on Multiple GPUs
huggingface.co · 1,096 words · saved by 1 readers
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
Multi-GPU setups are effective for accelerating training and fitting large models in memory that otherwise wouldn’t fit on a single GPU. It relies on parallelizing the workload across GPUs. There are several types of parallelism such as data parallelism, tensor parallelism, pipeline parallelism, and model parallelism. Each type of parallelism splits the workload differently, whether it’s the data or the model. This guide will discuss the various parallelism methods, combining them, and choosing an appropriate strategy for your setup. For more details about distributed training, refer to the…
related reading
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- How To Scale Your Modeljax-ml.github.io
- ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- 1910.02054v3arxiv.org
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com
- irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
- Breadth-First Pipeline Parallelismarxiv.org