16 questions about reinforcement learning

AlphaProof constructed a large curriculum of tasks by randomly misformalizing math olympiad problems; creating many problems of varying difficulty, from trivial to full math olympiad difficulty. This is a beautiful idea for building a curriculum; I dub it “task mutations”. Do task mutations work on other domains? Getting RL to scale from 0% to 1% is a big deal. But what about getting from 99% to 100%? These problems seem symmetrical to me; whether you are almost always wrong or almost always correct, you’re basically not getting any reward either way. Are the RL updates on correct samples reinforcing correct reasoning in a way that helps even when the model is getting everything correct, or is naive RL just limited as a method for getting to 100% reliability, in the same way it doesn’t work to get from 0% to 1%? Is there a canonical best method for getting out of the zero-reward regime, or is it always ad-hoc? The amount of information in a typical RLVR episode is at most 1 bit (we lea

AlphaProof constructed a large curriculum of tasks by randomly misformalizing math olympiad problems; creating many problems of varying difficulty, from trivial to full math olympiad difficulty. This is a beautiful idea for building a curriculum; I dub it “task mutations”. Do task mutations work on other domains? Getting RL to scale from 0% to 1% is a big deal. But what about getting from 99% to 100%? These problems seem symmetrical to me; whether you are almost always wrong or almost always correct, you’re basically not getting any reward either way. Are the RL updates on correct samples…

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Reasoning in General Domains without Verifiersarxiv.org
• Reinforcement Learning With Verifiable Rewards: How Data and Verifiers Shape RLVRsnorkel.ai
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• RLHF | John Lambertjohnwlambert.github.io
• [2412.14135] Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivearxiv.org
• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• [2510.13651] What is the objective of reasoning with reinforcement learning?arxiv.org