Tsinghua paper: Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — LessWrong

arXiv | project page | Authors: Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, Gao Huang This paper from Tsinghua find that RL on verifiable rewards (RLVR) just increases the frequency at which capabilities are sampled, rather than giving a base model new capabilities. To do this, they compare pass@k scores between a base model and an RLed model. Recall that pass@k is the percentage of questions a model can solve at least once given k attempts at each question. Main result: On a math benchmark, an RLed model (yellow) has much better raw score / pass@1 than the base model (black), but lower pass@256! The authors say that RL prunes away reasoning pathways from the base model, but sometimes reasoning pathways that are rarely sampled end up being useful for solving the problem. So RL “narrows the reasoning boundary”— the region of problems the model is capable of solving sometimes. Thanks to @Vladimir_Nesov for mentioning this paper here. Excepting artificia

x Tsinghua paper: Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? — LessWrong Language Models (LLMs) AI Frontpage 70 Tsinghua paper: Does RL Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? by Thomas Kwa 5th May 2025 3 min read 22 70 This is a linkpost for https://arxiv.org/abs/2504.13837 arXiv | project page | Authors: Yang Yue , Zhiqi Chen , Rui Lu , Andrew Zhao , Zhaokai Wang , Yang Yue , Shiji Song , Gao Huang This paper from Tsinghua find that RL on verifiable rewards (RLVR) just increases the frequency at which capabilities are sampled, ra

related reading

• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org
• DeepSeek-R1arxiv.org
• Limit of RLVRlimit-of-rlvr.github.io
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Reasoning in General Domains without Verifiersarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• the-illusion-of-thinking.pdfml-site.cdn-apple.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• The Invisible Leash: Why RLVR May Not Escape Its Originarxiv.org
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
• Reasoning Models Reason Well, Until They Don'tarxiv.org