Massively Scale Your Deep Learning Training with NCCL 2.4 | NVIDIA Technical Blog
Imagine using tens of thousands of GPUs to train your neural network. Using multiple GPUs to train neural networks has become quite common with all deep learning frameworks, providing optimized…
Massively Scale Your Deep Learning Training with NCCL 2.4 | NVIDIA Technical Blog Technical Blog Subscribe Related Resources Data Science Massively Scale Your Deep Learning Training with NCCL 2.4 Feb 04, 2019 By Sylvain Jeaugey Like Discuss (1) L T F R E AI-Generated Summary Like Dislike NCCL 2.4 introduces double binary trees, which offer full bandwidth and logarithmic latency for allreduce operations, enabling good performance on small and medium size operations. The double binary tree algorithm significantly improves latency, with up to 180x improvement at 24,576 GPUs on the Summit supercom
related reading
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklogsiboehm.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- How to Think About GPUs | How To Scale Your Modeljax-ml.github.io
- Multi-Datacenter Training: OpenAI's Ambitious Plan To Beat Google's Infrastructuresemianalysis.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- Updating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL - LMSYS Orglmsys.org
- Network and Storage Benchmarks for LLM Training on the Cloud | Henry Zhumaknee.github.io
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Optimizing a Layer Normalization Kernel with CUDA: a Worklogaryagxr.com
- Get Started With Deep Learning Performance - NVIDIA Docsdocs.nvidia.com