[2510.13651] What is the objective of reasoning with reinforcement learning?
Abstract:We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.
What is the objective of reasoning with reinforcement learning? Damek Davis∗ Benjamin Recht† October 16, 2025 arXiv:2510.13651v1 [cs.LG] 15 Oct 2025 Abstract We show that several popular algorithms for reinforcement…
saved by
related reading
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- From REINFORCE to Dr. GRPOlancelqf.github.io
- [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
- The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
- RLHF | John Lambertjohnwlambert.github.io
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- [2606.10346] Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningarxiv.org
- [2605.03327] DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentarxiv.org
- A Gallery of Methods Beyond RL — Part I: Sampling Methodsshengyu-feng.github.io