LLMs Can Self-Improve

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

L ARGE L ANGUAGE M ODELS C AN S ELF -I MPROVE Jiaxin Huang1∗ Shixiang Shane Gu2 Le Hou2† Yuexin Wu2 Xuezhi Wang2 Hongkun Yu2 Jiawei Han1 1 University of Illinois at Urbana-Champaign 2 Google 1 {jiaxinh3, hanj}@illinois.edu 2 {shanegu, lehou, crickwu, xuezhiw, hongkuny}@google.com…

related reading

• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Training Large Language Models to Reason in a Continuous Latent Spacearxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• [2603.07267] How to Steal Reasoning Without Reasoning Tracesarxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• 2401.10020.pdfarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2203.14465] STaR: Bootstrapping Reasoning With Reasoningarxiv.org