[2605.12484] Learning, Fast and Slow: Towards LLMs That Adapt Continually

Abstract:Large language models (LLMs) are trained for downstream tasks by updating their parameters (e.g., via RL). However, updating parameters forces them to absorb task-specific information, which can result in catastrophic forgetting and loss of plasticity. In contrast, in-context learning with fixed LLM parameters can cheaply and rapidly adapt to task-specific requirements (e.g., prompt optimization), but cannot by itself typically match the performance gains available through updating LLM parameters. There is no good reason for restricting learning to being in-context or in-weights. Moreover, humans also likely learn at different time scales (e.g., System 1 vs 2). To this end, we introduce a fast-slow learning framework for LLMs, with model parameters as "slow" weights and optimized context as "fast" weights. These fast "weights" can learn from textual feedback to absorb the task-specific information, while allowing slow weights to stay closer to the base model and persist general reasoning behaviors. Fast-Slow Training (FST) is up to 3x more sample-efficient than only slow learning (RL) across reasoning tasks, while consistently reaching a higher performance asymptote. Moreover, FST-trained models remain closer to the base LLM (up to 70% less KL divergence), resulting in less catastrophic forgetting than RL-training. This reduced drift also preserves plasticity: after training on one task, FST trained models adapt more effectively to a subsequent task than parameter-only trained models. In continual learning scenarios, where task domains change on the fly, FST continues to acquire each new task while parameter-only RL stalls.

Learning, Fast and Slow: Towards LLMs That Adapt Continually Rishabh Tiwari∗ 1,4 Kusha Sareen∗ 2 Lakshya A Agrawal∗ 1 Joseph E. Gonzalez 1 Matei Zaharia 1 Kurt Keutzer 1 Inderjit S Dhillon 3 Rishabh Agarwal† 2,5 Devvrit Khatri† 3,6 1 2 3…

saved by

• Yixiong Hao

related reading

• [2510.27062] Consistency Training Helps Stop Sycophancy and Jailbreaksarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• The Continual Learning Problemjessylin.com
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Self-Adapting Language Modelsarxiv.org
• [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org
• Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org