How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blog

The official Stanford AI Lab blog

In this post, we provide a Bayesian inference framework for in-context learning in large language models like GPT-3 and show empirical evidence for our framework, highlighting the differences from traditional supervised learning. This blog post primarily draws from the theoretical framework for in-context learning from An Explanation of In-context Learning as Implicit Bayesian Inference 1 and experiments from Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? 2 . TL;DR – In-context learning is a mysterious emergent behavior in large language models (LMs) where the LM p

saved by

• Stephen Lu
• abinaya dinesh
• Hangyul Lyna Kim

related reading

• In-context Learning and Induction Headstransformer-circuits.pub
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?arxiv.org
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• Learning without training: The implicit dynamics of in-context learningarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• What Can Transformers Learn In-Context?arxiv.org
• Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
• rethinking the role of demonstrations: what makes in-context learning work?arxiv.org
• You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
• In-context learning creates task vectorsarxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Large Language Model: world models or surface statistics?thegradient.pub