Efficient LLM Finetuning with Unsloth | Modal Docs

Training large language models is an incredibly compute-hungry process. Open-source LLMs often require many GBs (or in extreme cases, one TB!) of VRAM just to fit in memory. Finetuning models requires even more memory; a common estimate for naive finetuning puts the VRAM requirements at roughly 4.2x the original model size: 1x for model weights + 1x for gradients + 2x for optimizer state + 20% for activations. Parameter efficient methods like LoRA can improve matters significantly, since this estimate now applies to just the LoRA modules’ weights, rather than the entire model’s. Further gains can be made with quantization of each of the components mentioned above, but doing so requires quantization-aware training, which can be tricky to combine with methods like LoRA.

Efficient LLM Finetuning with Unsloth Training large language models is an incredibly compute-hungry process. Open-source LLMs often require many GBs (or in extreme cases, one TB !) of VRAM just to fit in memory. Finetuning models requires even more memory; a common estimate for naive finetuning puts the VRAM requirements at roughly 4.2x the original model size: 1x for model weights + 1x for gradients + 2x for optimizer state + 20% for activations. Parameter efficient methods like LoRA can improve matters significantly, since this estimate now applies to just the LoRA modules’ weights, rather

saved by

• Sheikh Abdur Raheem Ali

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Fine-tuning LLMs Guide | Unsloth Documentationdocs.unsloth.ai
• Finetune LLMs on your own consumer hardware using tools from PyTorch and Hugging Face ecosystem – PyTorchpytorch.org
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• 2305.14314arxiv.org
• Memory Efficient RL | Unsloth Documentationdocs.unsloth.ai
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Tutorial: How to Finetune Llama-3 and Use In Ollama | Unsloth Documentationdocs.unsloth.ai
• Parameter-Efficient LLM Finetuning With Low-Rank Adaptation (LoRA) - Lightning AIlightning.ai
• In-depth guide to fine-tuning LLMs with LoRA and QLoRA | Mercity Researchmercity.ai
• Efficient LLM inferencefinbarrtimbers.substack.com