Advanced RL Documentation | Unsloth Documentation
num_iterations (int, default 1): PPO epochs per batch (μ in the algorithm). Replays data within each gradient accumulation step; e.g., 2 = two forward passes per accumulation step. epsilon (float, default 0.2): Clipping value for token-level log-prob ratios (typical ratio range ≈ [-1.2, 1.2] with default ε). delta (float, optional): Enables upper clipping bound for two-sided GRPO when set. If None, standard GRPO clipping is used. Recommended > 1 + ε when enabled (per INTELLECT-2 report). epsilon_high (float, optional): Upper-bound epsilon; defaults to epsilon if unset. DAPO recommends 0.28. importance_sampling_level (“token” | “sequence”, default "token"): "token": raw per-token ratios (one weight per token). "sequence": average per-token ratios to a single sequence-level ratio. GSPO shows sequence-level sampling often gives more stable training for sequence-level rewards. reward_weights (list[float], optional): One weight per reward. If None, all weights = 1.0. scale_rewards (str|bool
Advanced Reinforcement Learning Documentation | Unsloth Documentation 🇺🇸 English Get Started 🦥 Homepage 🔮 Models ⭐ Beginner? 📒 Unsloth Notebooks 📥 Installation 🧬 Fine-tuning Guide 💡 Reinforcement Learning 🌀 7x Longer Context RL 👁️🗨️ Vision RL 🎱 FP8 RL ⚡ Tutorial: GRPO Training 🧩 Advanced RL Docs GSPO RL RL Reward Hacking ⁉️ FP16 vs BF16 for RL Memory Efficient RL 🏆 DPO, ORPO, KTO New Unsloth for AMD 🦥 Introducing Unsloth Studio Unsloth Updates Models Complete LLM Directory GLM-5.2 💜 Qwen3.6 ✨ Gemma 4 DeepSeek-V4 Inkling 🪽 Run MTP Models DiffusionGemma 💜 Qwen3.5 Basics Unslot
Explore this link on the map →related reading
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- State of RL for reasoning LLMs | A. Weersaweers.de
- The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
- RLHF Bookrlhfbook.com
- Tutorial: Train your own Reasoning model with GRPO | Unsloth Documentationdocs.unsloth.ai
- Memory Efficient RL | Unsloth Documentationdocs.unsloth.ai
- A Guide to Reinforcement Learning Post-Training for LLMs: PPO, DPO, GRPO, and Beyondhuggingface.co
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- PPO for LLMs: A Guide for Normal Peoplecameronrwolfe.substack.com
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
- Why GRPO is Important and How it Worksghost.oxen.ai