Pretraining parallelisms and failed runs — Flashcards
flashcards.dwarkesh.com · 989 words · saved by 1 readers
Why pretraining runs fail, and the chain of problems and fixes behind FSDP / pipeline / tensor parallelism.
Pretraining parallelisms and failed runs — Flashcards Why pretraining runs fail Expand all 01 Why is breaking causality in training so bad? The model just learns the trivial solution of copying the answer from the output, and the gradient carries no useful signal. 02 What is expert choice, and why does it break causality? You split the tokens across experts by which tokens each expert most strongly prefers, so every expert ends up with roughly the same number of tokens. But which expert token n n n gets allocated to can depend on which expert token n + k n + k n + k might be routed to. 03 What
saved by
related reading
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Visualizing 6D Mesh Parallelism · mainmain-horse.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- How To Scale Your Modeljax-ml.github.io
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai