[2502.00212] STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving

Abstract:A fundamental challenge in formal theorem proving by LLMs is the lack of high-quality training data. Although reinforcement learning or expert iteration partially mitigates this issue by alternating between LLM generating proofs and finetuning them on correctly generated ones, performance quickly plateaus due to the scarcity of correct proofs (sparse rewards). To keep improving the models with limited data, we draw inspiration from mathematicians, who continuously develop new results, partly by proposing novel conjectures or exercises (which are often variants of known results) and attempting to solve them. We design the Self-play Theorem Prover (STP) that simultaneously takes on two roles, conjecturer and prover, each providing training signals to the other. The conjecturer is trained iteratively on previously generated conjectures that are barely provable by the current prover, which incentivizes it to generate increasingly challenging conjectures over time. The prover attempts to prove the conjectures with standard expert iteration. We evaluate STP with both Lean and Isabelle formal versifiers. With 19.8 billion tokens generated during the training in Lean, STP proves 26.3% of the statements in the LeanWorkbook dataset, doubling the previous best result of 13.2% achieved through expert iteration. The final model achieves state-of-the-art performance among whole-proof generation methods on miniF2F-test (61.7%, pass@3200), Proofnet-test (23.1%, pass@3200) and PutnamBench (8/644, pass@3200).

STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving Kefan Dong Tengyu Ma Stanford University Stanford University kefandong@stanford.edu tengyuma@stanford.edu arXiv:2502.00212v4 [cs.LG] 21 Mar 2025…

related reading

• Scaling Self-Play with Self-Guidancearxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Reasoning in General Domains without Verifiersarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• 2009.03393arxiv.org
• [2203.14465] STaR: Bootstrapping Reasoning With Reasoningarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2510.01346] Aristotle: IMO-level Automated Theorem Provingarxiv.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Let's Verify Step by Steparxiv.org