Everything about Distributed Training and Efficient Finetuning | Sumanth's Personal Website
A deep dive into distributed training and efficient finetuning - DeepSpeed ZeRO, FSDP, practical guidelines and gotchas with multi-GPU and multi-node training
Everything about Distributed Training and Efficient Finetuning A deep dive into distributed training and efficient finetuning - DeepSpeed ZeRO, FSDP, practical guidelines and gotchas with multi-GPU and multi-node training Last updated on Jan 19, 2024 40 min read There's been an insane amount of interest in large language models (LLMs) these days, with a very special open source community of hackers figuring out the best way to finetune, serve and run inference on consumer-grade hardware. A number of excellent open-source codebases have popped up to meet these needs, notably FastChat , Axolotl
Explore this link on the map →related reading
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- How To Scale Your Modeljax-ml.github.io
- DeepSpeed ZeRO++: A leap in speed for LLM and chat model training with 4X less communication - Microsoft Researchmicrosoft.com
- Fully Sharded Data Parallel: faster AI training with fewer GPUs Engineering at Meta -engineering.fb.com
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- 5D parallelism in LLM training - gdymind's Bloggdymind.com
- 👨👩👧👦 Distributed Training - Composerdocs.mosaicml.com
- Paradigms of Parallelism | Colossal-AIcolossalai.org
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
- Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
- Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org