Breadth-First Pipeline Parallelism
arxiv.org · 6,515 words · saved by 1 readers
N/A
B READTH -F IRST P IPELINE PARALLELISM Joel Lamy-Poirier 1 A BSTRACT We introduce Breadth-First Pipeline Parallelism, a novel training schedule which optimizes the combination of pipeline and data parallelism. Breadth-First Pipeline Parallelism lowers training time, cost and memory usage arXiv:2211.05953v2 [cs.DC] 6…
related reading
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Parallel CPU-GPU Execution for LLM Inference on Constrained GPUsarxiv.org
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Reducing Activation Recomputationarxiv.org
- irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
- [1909.08053] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismarxiv.org
- 2305.19370arxiv.org