Paradigms of Parallelism | Colossal-AI
colossalai.org · 1,338 words · saved by 1 readers
Author: Shenggui Li, Siqi Mai
On this page Paradigms of Parallelism Author: Shenggui Li, Siqi Mai Introduction With the development of deep learning, there is an increasing demand for parallel training. This is because that model and datasets are getting larger and larger and training time becomes a nightmare if we stick to single-GPU training. In this section, we will provide a brief overview of existing methods to parallelize training. If you wish to add on to this post, you may create a discussion in the GitHub forum . Data Parallel Data parallel is the most common form of parallelism due to its simplicity. In data
saved by
related reading
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Parallelism methods · Hugging Facehuggingface.co
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- How To Scale Your Modeljax-ml.github.io
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
- 1910.02054v3arxiv.org
- irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
- Breadth-First Pipeline Parallelismarxiv.org
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelsarxiv.org