Efficient LLM Finetuning with Unsloth | Modal Docs

Training large language models is an incredibly compute-hungry process. Open-source LLMs often require many GBs (or in extreme cases, one TB!) of VRAM just to fit in memory. Finetuning models requires even more memory; a common estimate for naive finetuning puts the VRAM requirements at roughly 4.2x the original model size: 1x for model weights + 1x for gradients + 2x for optimizer state + 20% for activations. Parameter efficient methods like LoRA can improve matters significantly, since this estimate now applies to just the LoRA modules’ weights, rather than the entire model’s. Further gains can be made with quantization of each of the components mentioned above, but doing so requires quantization-aware training, which can be tricky to combine with methods like LoRA.

Efficient LLM Finetuning with Unsloth Training large language models is an incredibly compute-hungry process. Open-source LLMs often require many GBs (or in extreme cases, one TB !) of VRAM just to fit in memory. Finetuning models requires even more memory; a common estimate for naive finetuning puts the VRAM requirements at roughly 4.2x the original model size: 1x for model weights + 1x for gradients + 2x for optimizer state + 20% for activations. Parameter efficient methods like LoRA can improve matters significantly, since this estimate now applies to just the LoRA modules’ weights, rather

saved by

• Sheikh Abdur Raheem Ali

related reading

• Fine-tuning LLMs Guide | Unsloth Documentationdocs.unsloth.ai
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Tutorial: How to Finetune Llama-3 and Use In Ollama | Unsloth Documentationdocs.unsloth.ai
• Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
• The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities (Version 1.0)arxiv.org
• Fine-tuning a LLM on my blog posts | Didier Lopesdidierlopes.com
• GitHub - bobazooba/xllm: 🦖 X—LLM: Cutting Edge & Easy LLM Finetuning · GitHubgithub.com
• Fine-Tuning Llama-2: Tailoring Models to Unique Applicationsanyscale.com
• Unsloth update: Mistral support + moreunsloth.ai
• The Llama Hitchiking Guide to Local LLMs – hackerllamaosanseviero.github.io