LLMs Can Self-Improve

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

L ARGE L ANGUAGE M ODELS C AN S ELF -I MPROVE Jiaxin Huang1∗ Shixiang Shane Gu2 Le Hou2† Yuexin Wu2 Xuezhi Wang2 Hongkun Yu2 Jiawei Han1 1 University of Illinois at Urbana-Champaign 2 Google 1 {jiaxinh3, hanj}@illinois.edu 2 {shanegu, lehou, crickwu, xuezhiw, hongkuny}@google.com…

related reading

• DeepSeek-R1arxiv.org
• Self-Adapting Language Modelsarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Can LLMs Critique and Iterate on Their Own Outputs? | Eric Jangevjang.com
• [2203.14465] STaR: Bootstrapping Reasoning With Reasoningarxiv.org
• 2401.10020.pdfarxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• LLM Post-Training: A Deep Dive into Reasoning Large Language Modelsarxiv.org
• Productizing Large Language Modelsblog.replit.com
• Do Large Language Models (LLMs) reason? | Shapedshaped.ai
• 2402.07927arxiv.org
• [2504.13837] Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?arxiv.org