Model Parallelism Overview
We will first discuss in depth various 1D parallelism techniques and their pros and cons and then look at how they can be combined into 2D and 3D parallelism to enable an even faster training and to support even bigger models. Various other powerful alternative approaches will be presented. While the main concepts most likely will apply to any other framework, this article is focused on PyTorch-based implementations. Two main approaches are used to enable training and inferring models that are bigger than the accelerator's memory: The following is the brief description of the main concepts that will be described later in depth in this document. The introduction sections of this paper is probably one of the best explanations I have found on most common parallelism techniques Breadth-First Pipeline Parallelism. Most users with just 2 GPUs already enjoy the increased training speed up thanks to DataParallel (DP) and DistributedDataParallel (DDP) that are almost trivial to use. This is a b
Model Parallelism Moved to Model Parallelism .
Explore this link on the map →related reading
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- How To Scale Your Modeljax-ml.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Making Deep Learning go Brrrr From First Principleshorace.io
- How to Parallelize Deep Learning on GPUs Part 1/2: Data Parallelism - Tim Dettmerstimdettmers.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Visualizing 6D Mesh Parallelism · mainmain-horse.github.io
- PiTorch: ML on Baremetal Raspberry Pis | projectsmasonjwang.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly