[2510.13651] What is the objective of reasoning with reinforcement learning?
Abstract:We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.
What is the objective of reasoning with reinforcement learning? Damek Davis∗ Benjamin Recht† October 16, 2025 arXiv:2510.13651v1 [cs.LG] 15 Oct 2025 Abstract We show that several popular algorithms for reinforcement…
saved by
related reading
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- State of RL for reasoning LLMs | A. Weersaweers.de
- RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- DeepSeek-R1arxiv.org
- RLHF | John Lambertjohnwlambert.github.io
- RLHF Bookrlhfbook.com
- From REINFORCE to Dr. GRPOlancelqf.github.io
- A Gallery of Methods Beyond RL — Part I: Sampling Methodsshengyu-feng.github.io
- Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io