[2506.10947] Spurious Rewards: Rethinking Training Signals in RLVR

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2506.10947] Spurious Rewards: Rethinking Training Signals in RLVR Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:2506.10947 (cs) [Submitted on 12 Jun 2025 ( v1 ), last revised 25 Feb 2026 (this version, v2)] Title: Spurious Rewards: Rethinking Training Signals in RLVR Authors: Rulin Shao , Shuyue Stella Li , Rui Xin , Scott Geng , Yiping Wang , Sewoong Oh , Simon Shaolei Du , Nathan Lambert , Sewon Min , Ranjay Krishna , Yulia Tsvetkov , Hannaneh Haj

related reading

• State of RL for reasoning LLMs | A. Weersaweers.de
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• DeepSeek-R1arxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectoriesarxiv.org
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• VRPRM: Process Reward Modeling via Visual Reasoningarxiv.org
• Features as Rewards: Using Interpretability to Reduce Hallucinationsgoodfire.ai
• Understanding R1-Zero-Like Training: A Critical Perspectivearxiv.org
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• The State of Reinforcement Learning for LLM Reasoningsebastianraschka.com