[2506.10947] Spurious Rewards: Rethinking Training Signals in RLVR

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs. arXiv Operational Status Get status notifications via email or slack

[2506.10947] Spurious Rewards: Rethinking Training Signals in RLVR Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:2506.10947 (cs) [Submitted on 12 Jun 2025 ( v1 ), last revised 25 Feb 2026 (this version, v2)] Title: Spurious Rewards: Rethinking Training Signals in RLVR Authors: Rulin Shao , Shuyue Stella Li , Rui Xin , Scott Geng , Yiping Wang , Sewoong Oh , Simon Shaolei Du , Nathan Lambert , Sewon Min , Ranjay Krishna , Yulia Tsvetkov , Hannaneh Haj

related reading

• [2401.05566] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingarxiv.org
• [2605.12474] Reward Hacking in Rubric-Based Reinforcement Learningarxiv.org
• Reasoning in General Domains without Verifiersarxiv.org
• [2503.10965] Auditing language models for hidden objectivesarxiv.org
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainsarxiv.org
• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
• [2512.18552] Toward Training Superintelligent Software Agents through Self-Play SWE-RLarxiv.org
• [2406.11717] Refusal in Language Models Is Mediated by a Single Directionarxiv.org
• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• [2602.15515] The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probesarxiv.org
• [2606.10346] Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningarxiv.org
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org