Bug Fixes in LLM Training - Gradient Accumulation

Unsloth's Gradient Accumulation fix solves critical errors in LLM Training.

Bug Fixes in LLM Training - Gradient Accumulation unsloth Models Blog Unsloth Studio✨ Docs Blog Bugs in LLM Training - Gradient Accumulation Fix Oct 15, 2024 • By Daniel & Michael Oct 15, 2024 • By Daniel & Michael This past week, we've been fixing a universal issue in Gradient Accumulation that negatively impacts everyone's training, pre-training & finetuning runs for sequence models like LLMs. Unsloth's Gradient Accumulation fix ensures training runs and loss calculations are performed accurately and correctly. The goal of gradient accumulation is to mimic full batch training with reduced VR

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• microgptkarpathy.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Efficient LLM Finetuning with Unsloth | Modal Docsmodal.com
• Fine-tuning LLMs Guide | Unsloth Documentationdocs.unsloth.ai
• From REINFORCE to Dr. GRPOlancelqf.github.io
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• Unsloth update: Mistral support + moreunsloth.ai
• RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
• GitHub - linkedin/Liger-Kernel: Efficient Triton Kernels for LLM Training · GitHubgithub.com