How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blog

The official Stanford AI Lab blog

In this post, we provide a Bayesian inference framework for in-context learning in large language models like GPT-3 and show empirical evidence for our framework, highlighting the differences from traditional supervised learning. This blog post primarily draws from the theoretical framework for in-context learning from An Explanation of In-context Learning as Implicit Bayesian Inference 1 and experiments from Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? 2 . TL;DR – In-context learning is a mysterious emergent behavior in large language models (LMs) where the LM p

saved by

• abinaya dinesh
• Hangyul Lyna Kim

related reading

• In-context Learning and Induction Headstransformer-circuits.pub
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• [2301.00234] A Survey on In-context Learningarxiv.org
• Why We Need Continual Learning | Andreessen Horowitza16z.com
• GenAI Handbookgenai-handbook.github.io
• [2212.07677] Transformers learn in-context by gradient descentarxiv.org
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• [2311.06668] In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringarxiv.org
• NL.pdfabehrouz.github.io
• [2309.01809] Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
• [2506.06266] Cartridges: Lightweight and general-purpose long context representations via self-studyarxiv.org