[2507.16003] Learning without training: The implicit dynamics of in-context learning

Abstract:One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer according to the context. We argue through theoretical analysis and experimentation that this simple mechanism may help explain why LLMs demonstrate capabilities of in-context learning, beyond what is captured during training. Specifically, we show that a standard forward pass with context is mathematically equivalent to a forward pass without context but with the MLP weights updated by a minimal low-rank update representing the context.

View PDF HTML (experimental) Abstract:One of the most striking features of Large Language Models (LLMs) is their ability to learn in-context. Namely at inference time an LLM is able to learn new patterns without any additional weight update when these patterns are presented in the form of examples in the prompt, even if these patterns were not seen during training. The mechanisms through which this can happen are still largely unknown. In this work, we show that the stacking of a self-attention layer with an MLP allows the transformer block to implicitly modify the weights of the MLP layer…

saved by

• Abhi Upadhyay

related reading

• In-context Learning and Induction Headstransformer-circuits.pub
• How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogai.stanford.edu
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?arxiv.org
• NL.pdfabehrouz.github.io
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2212.07677] Transformers learn in-context by gradient descentarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencearxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• A History of Large Language Modelsgregorygundersen.com
• Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org