1910.02054v3
arxiv.org · 8,777 words · saved by 1 readers
N/A
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Samyam Rajbhandari∗ , Jeff Rasley∗, Olatunji Ruwase, Yuxiong He arXiv:1910.02054v3 [cs.LG] 13 May 2020 {samyamr, jerasley, olruwase, yuxhe}@microsoft.com Abstract Large deep learning models offer significant accuracy gains, but training billions to trillions…
saved by
related reading
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- DeepSpeed ZeRO++: A leap in speed for LLM and chat model training with 4X less communication - Microsoft Researchmicrosoft.com
- Training Models Too Big for One GPU | ZeRO & FSDPyoutube.com
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
- [1909.08053] Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismarxiv.org
- 2305.14314arxiv.org
- Reducing Activation Recomputationarxiv.org
- [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org