✳flâneur — a map of the web's best reading

How to train a model on 10k H100 GPUs?

soumith.ch · 920 words · saved by 1 readers

I quickly jotted down what I think is fairly common knowledge among engineers working on large-scale training. There's three parts. Keep parallelizing until you are able to use all GPUs well, with maximum utilization. When you need to communicate among GPUs, try to start communication as soon as you can: Communicating large amounts of state (gradients, optimizer state) across multiple nodes is complicated. with Sync SGD, you have to communicate this state in a burst, as quickly as you can. we might have multiple layers of switches, and have RDMA (ability to copy GPU memory directly to NIC, bypassing CPU ram entirely), and have frontend and backend NICs (frontend connects to storage like NFS, backend connects GPUs to other GPUs in cluster). So, it's important to leverage all this info when running communication collectives like all-reduce or scatter/gather. All-reduce for example can be done algorithmically in log(n) if you tree-reduce; and the constant factors that change based on the

**How to train a model on 10k H100 GPUs?** A quick note summarizing common knowledge among the large-scale training cohort Oct 2nd, 2024 [https://soumith.ch/blog.html](https://soumith.ch/blog.html) I realize I have absolutely no clue how you train a single model with e.g. 10k h100s. What is where, updated when, with what? - François Fleuret (@francoisfleuret) September 30, 2024 My friend Francois Fleuret asked the above. I quickly jotted down what I think is fairly common knowledge among engineers working on large-scale training. There's three parts. 1. Fitting as large of a network and as lar

Explore this link on the map →

related reading