How to Parallelize a Transformer for Training | How To Scale Your Model
jax-ml.github.io · saved by 1 readers
Here we discuss four main parallelism schemes used during LLM training: data parallelism, fully-sharded data parallelism (FSDP), tensor parallelism, and pipeline parallelism. For each, we calculate at what point we become bottlenecked by communication.