[2303.11366] Reflexion: Language Agents with Verbal Reinforcement Learning

Abstract:Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance.

[2303.11366] Reflexion: Language Agents with Verbal Reinforcement Learning Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Artificial Intelligence arXiv:2303.11366 (cs) [Submitted on 20 Mar 2023 ( v1 ), last revised 10 Oct 2023 (this version, v4)] Title: Reflexion: Language Agents with Verbal Reinforcement Learning Authors: Noah Shinn , Federico Cassano , Edward Berman , Ashwin Gopinath , Karthik Narasimhan , Shunyu Yao View a PDF of the paper titled Reflexion: Language Agents with

related reading

• 1b44b878bb782e6954cd888628510e90-Paper-Conference.pdfproceedings.neurips.cc
• Illustrating Reinforcement Learning from Human Feedback (RLHF)huggingface.co
• Can LLMs Critique and Iterate on Their Own Outputs? | Eric Jangevjang.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Language Models can Solve Computer Tasksarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• 2401.10020.pdfarxiv.org
• Noah Shinnnoahshinn.com