Noisy Data Breaks RLVR - by Daniel Kang - Daniel’s Substack

Reinforcement learning with verifiable rewards (RLVR) is a widely used post-training paradigm to improve the reasoning capabilities of LLMs.

Reinforcement learning with verifiable rewards (RLVR) is a widely used post-training paradigm to improve the reasoning capabilities of LLMs. However, creating the high-quality verifiable answers needed to train the model is labor-intensive and expensive, as highlighted by the boom of data labeling companies, such as Mercor. This creates a fundamental question for post-training: To what extent can RLVR, with algorithmic improvements, tolerate large-volume, noisy data? Recent literature promotes a counter-intuitive idea: RLVR is robust to noisy data. Prior work shows that training on “100%…

saved by

• Varun Goyal

related reading

• Putting Task Expertise into RL Achieves State-of-the-Art Performance on Text-to-SQLthinkingmachines.ai
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
• Reasoning in General Domains without Verifiersarxiv.org
• Reinforcement Learning With Verifiable Rewards: How Data and Verifiers Shape RLVRsnorkel.ai
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2602.05910] Chunky Post-Training: Data Driven Failures of Generalizationarxiv.org
• Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org