[2505.17373] Value-Guided Search for Efficient Chain-of-Thought Reasoning

Abstract:In this paper, we propose a simple and efficient method for value model training on long-context reasoning traces. Compared to existing process reward models (PRMs), our method does not require a fine-grained notion of "step," which is difficult to define for long-context reasoning models. By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model and apply it to DeepSeek models for improved performance with test-time compute scaling. We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods such as majority voting or best-of-n. Moreover, VGS significantly reduces the inference FLOPs required to achieve the same performance of majority voting. Our dataset, model and codebase are open-sourced.

Value-Guided Search for Efficient Chain-of-Thought Reasoning Kaiwen Wang*1 , Jin Peng Zhou*1 , Jonathan Chang*4 , Zhaolin Gao1 , Nathan Kallus1,3 , Kianté Brantley2 , and Wen Sun1 1 2 3…

saved by

• Katherine Driscoll

related reading

• DeepSeek-R1arxiv.org
• As Rocks May Think | Eric Jangevjang.com
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelsarxiv.org
• Explore | alphaXivalphaxiv.org
• o1 and Reasoning | AndoLogsblog.ando.ai
• DeepSeek R1's recipe to replicate o1 and the future of reasoning LMsinterconnects.ai
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• QwenLong-L1.5: Post-Training Recipe for Long-Context Reasoning and Memory Managementarxiv.org
• Stream of Search (SoS): Learning to Search in Languagearxiv.org
• Open-R1: a fully open reproduction of DeepSeek-R1huggingface.co