[2505.17373] Value-Guided Search for Efficient Chain-of-Thought Reasoning

Abstract:In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.

Value-Guided Search for Efficient Chain-of-Thought Reasoning Kaiwen Wang*1 , Jin Peng Zhou*1 , Jonathan Chang*4 , Zhaolin Gao1 , Nathan Kallus1,3 , Kianté Brantley2 , and Wen Sun1 1 2 3…

saved by

• Katherine Driscoll

related reading

• DeepSeek-R1arxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Stream of Search (SoS): Learning to Search in Languagearxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2604.18002] Neural Garbage Collection: Learning to Forget while Learning to Reasonarxiv.org
• [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasksarxiv.org
• [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyarxiv.org
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsarxiv.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org