[2605.12474] Reward Hacking in Rubric-Based Reinforcement Learning

Abstract:Reinforcement learning with verifiable rewards has enabled strong post-training gains in domains such as math and coding, though many open-ended settings rely on rubric-based rewards. We study reward hacking in rubric-based RL, where a policy is optimized against a training verifier but evaluated against a cross-family panel of three frontier judges, reducing dependence on any single evaluator. Our framework separates two sources of divergence: verifier failure, where the training verifier credits rubric criteria that reference verifiers reject, and rubric-design limitations, where even strong rubric-based verifiers favor responses that rubric-free judges rate worse overall. Across medical and science domains, weak verifiers produce large proxy-reward gains that do not transfer to the reference verifiers; exploitation grows over training and concentrates in recurring failures such as partial satisfaction of compound criteria, treating implicit content as explicit, and imprecise topical matching. Stronger verifiers substantially reduce, but do not eliminate, verifier exploitation. We also introduce a self-internalization gap, a verifier-free diagnostic based on policy log-probabilities, which tracks reference-verifier quality, detecting when the policy trained using the weak verifier stops improving. Finally, in our setting, stronger verification does not prevent reward hacking when the rubric leaves important failure modes unspecified: rubric-based verifiers prefer the RL checkpoint, while rubric-free judges prefer the base model. These disagreements coincide with gains concentrated in completeness and presence-based criteria, alongside declines in factual correctness, conciseness, relevance, and overall quality. Together, these results suggest that stronger verification reduces reward hacking, but does not by itself ensure that rubric gains correspond to broader quality gains.

[2605.12474] Reward Hacking in Rubric-Based Reinforcement Learning --> Computer Science > Artificial Intelligence arXiv:2605.12474 (cs) [Submitted on 12 May 2026] Title: Reward Hacking in Rubric-Based Reinforcement Learning Authors: Anas Mahmoud , MohammadHossein Rezaei , Zihao Wang , Anisha Gunjal , Bing Liu , Yunzhong He View a PDF of the paper titled Reward Hacking in Rubric-Based Reinforcement Learning, by Anas Mahmoud and 5 other authors View PDF HTML (experimental) Abstract: Reinforcement learning with verifiable rewards has enabled strong post-training gains in domains such as math and

saved by

• Su P

related reading

• [2605.10889] Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Whyarxiv.org
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainsarxiv.org
• Natural-emergent-misalignment-from-reward-hacking-paper.pdfassets.anthropic.com
• Training on Documents about Reward Hacking Induces Reward Hackingalignment.anthropic.com
• [2607.18966] Measuring Reward-Seeking via Contrastive Belief Updatesarxiv.org
• [2606.12016] Generalization Hacking: Models Can Game Reinforcement Learning by Preventing Behavioral Generalizationarxiv.org
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• [2506.10947] Spurious Rewards: Rethinking Training Signals in RLVRarxiv.org
• [2602.15515] The Obfuscation Atlas: Mapping Where Honesty Emerges in RLVR with Deception Probesarxiv.org
• Large Language Models Hack Rewards, and Societyarxiv.org