LLMs Can Self-Improve

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate "high-confidence" rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We sh

related reading

• LLM Powered Autonomous Agents | Lil'Loglilianweng.github.io
• DeepSeek-R1arxiv.org
• Self-Adapting Language Modelsarxiv.org
• Do Large Language Models (LLMs) reason? | Shapedshaped.ai
• [2309.16797] Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutionarxiv.org
• [2309.16797] Promptbreeder: Self-Referential Self-Improvement via Prompt Evolutionar5iv.labs.arxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• Teaching LLMs to reason like Bayesiansresearch.google
• Language Models can Solve Computer Tasksarxiv.org
• [2405.20974] SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalesarxiv.org
• [2310.08118] Can Large Language Models Really Improve by Self-critiquing Their Own Plans?arxiv.org