Pipeline-Parallelism: Distributed Training via Model Partitioning
Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory.Example: Huggingface’s BLOOM model is a 175B parameter...
Pipeline-Parallelism: Distributed Training via Model Partitioning Subscribe Si_Boehm October 2022 Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory. Example: Huggingface’s BLOOM model is a 175B parameter Transformer model. Storing the weights as bfloat16 requires 350GB, but the GPUs they used to train BLOOM ‘only’ have 80GB of memory, and training requires much more memory than just loading the model weights. So their final training was distributed across 384 GPUs. This is made possible by assigning different layers of the model to different
saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Breadth-First Pipeline Parallelismarxiv.org
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- How to Parallelize Deep Learning on GPUs Part 1/2: Data Parallelism - Tim Dettmerstimdettmers.com
- Parallelism methods · Hugging Facehuggingface.co
- ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu