How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon University
LLM RL typically operates in one of three exploration regimes: sharpening, chaining, or guided exploration; standard RL stays in the first two and plateaus on hard problems, even those in the training set. Mixing easy and hard data triggers interference, causing reward-rich tasks to drown out hard o
How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon University --> Input your search keywords and press Enter. Categories: Research Educational Categories: Research Educational machine learning reinforcement learning Research How to Explore to Scale RL Training of LLMs on Hard Problems? Authors Amrith Setlur by Amrith Setlur --> Affiliations Published November 26, 2025 DOI Figure 1. Three regimes of exploration: Current RL model can explore via: (1) sharpening : simply increases likelihood on traces it can sample with high prob
saved by
related reading
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- Q-learning is not yet scalableseohong.me
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- State of RL for reasoning LLMs | A. Weersaweers.de
- Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- Lil'Loglilianweng.github.io
- DeepSeek-R1arxiv.org
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com