Pipeline-Parallelism: Distributed Training via Model Partitioning
Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory.Example: Huggingface’s BLOOM model is a 175B parameter...
Pipeline-Parallelism: Distributed Training via Model Partitioning Subscribe Si_Boehm October 2022 Pipeline parallelism makes it possible to train large models that don’t fit into a single GPU’s memory. Example: Huggingface’s BLOOM model is a 175B parameter Transformer model. Storing the weights as bfloat16 requires 350GB, but the GPUs they used to train BLOOM ‘only’ have 80GB of memory, and training requires much more memory than just loading the model weights. So their final training was distributed across 384 GPUs. This is made possible by assigning different layers of the model to different
saved by
related reading
- How To Scale Your Modeljax-ml.github.io
- Making Deep Learning go Brrrr From First Principleshorace.io
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- Breadth-First Pipeline Parallelismarxiv.org
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- How to Think About TPUs | How To Scale Your Modeljax-ml.github.io
- Parallelism methods · Hugging Facehuggingface.co
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- irhum.github.io - Tensor Parallelism with jax.pjitirhum.github.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly