✳flâneur — a map of the web's best reading
Pretraining parallelisms and failed runs — Flashcards
flashcards.dwarkesh.com · 989 words · saved by 1 readers
Why pretraining runs fail, and the chain of problems and fixes behind FSDP / pipeline / tensor parallelism.
Pretraining parallelisms and failed runs — Flashcards Why pretraining runs fail Expand all 01 Why is breaking causality in training so bad? The model just learns the trivial solution of copying the answer from the output, and the gradient carries no useful signal. 02 What is expert choice, and why does it break causality? You split the tokens across experts by which tokens each expert most strongly prefers, so every expert ends up with roughly the same number of tokens. But which expert token n n n gets allocated to can depend on which expert token n + k n + k n + k might be routed to. 03 What
Explore this link on the map →saved by
related reading
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Visualizing 6D Mesh Parallelism · mainmain-horse.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- How To Scale Your Modeljax-ml.github.io
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Websitesumanthrh.com