Inside TPU and GPU Clusters: The Anatomy of Collective Communication

A deep dive into all-gather, reduce-scatter, all-reduce, and all-to-all communication on TPUs and GPUs.

saved by

• Brady Gho
• Ishaan Panigrahi
• Chris Shi