Pipeline-Parallelism: Distributed Training via Model Partitioning
Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory.Example: Huggingface’s BLOOM model is a 175B parameter...
Pipeline-Parallelism: Distributed Training via Model Partitioning Subscribe Si_Boehm October 2022 Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory. Example: Huggingface’s BLOOM model is a 175B parameter Transformer model. Storing the weights as bfloat16 requires 350GB, but the GPUs they used to train BLOOM ‘only’ have 80GB of memory, and training requires much more memory than just loading the model weights. So their final training was distributed across 384 GPUs. This is made possible by assigning different layers of the model to different
Explore this link on the map →saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- How to Parallelize Deep Learning on GPUs Part 1/2: Data Parallelism - Tim Dettmerstimdettmers.com
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- ml-engineering/model-parallelism at master · stas00/ml-engineering · GitHubgithub.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Visualizing 6D Mesh Parallelism · mainmain-horse.github.io