[2510.13651] What is the objective of reasoning with reinforcement learning?

Abstract:We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.

What is the objective of reasoning with reinforcement learning? We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.

saved by

• Jaisel Singh

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• DeepSeek-R1arxiv.org
• The State of Reinforcement Learning for LLM Reasoningsebastianraschka.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Reward is not the optimization target — LessWronglesswrong.com
• RLHF Bookrlhfbook.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Why GRPO is Important and How it Worksghost.oxen.ai
• [2605.03327] DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentarxiv.org
• From REINFORCE to Dr. GRPOlancelqf.github.io