Pretraining parallelisms and failed runs — Flashcards
flashcards.dwarkesh.com · 989 words · saved by 1 readers
Why pretraining runs fail, and the chain of problems and fixes behind FSDP / pipeline / tensor parallelism.
Pretraining parallelisms and failed runs — Flashcards Why pretraining runs fail Expand all 01 Why is breaking causality in training so bad? The model just learns the trivial solution of copying the answer from the output, and the gradient carries no useful signal. 02 What is expert choice, and why does it break causality? You split the tokens across experts by which tokens each expert most strongly prefers, so every expert ends up with roughly the same number of tokens. But which expert token n n n gets allocated to can depend on which expert token n + k n + k n + k might be routed to. 03 What
saved by
related reading
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- How To Scale Your Modeljax-ml.github.io
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- Making Deep Learning go Brrrr From First Principleshorace.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- Visualizing 6D Mesh Parallelism · mainmain-horse.github.io
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com