[2202.12837] Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance, consistently over 12 different models including GPT-3. Instead, we find that other aspects of the demonstrations are the key drivers of end task performance, including the fact that they provide a few examples of (1) the label space, (2) the distribution of the input text, and (3) the overall format of the sequence. Together, our analysis provides a new way of understanding how and why in-context learning works, while opening up new questions about how much can be learned from large language models through inference alone.
View PDF HTML (experimental) Abstract:Large language models (LMs) are able to in-context learn -- perform a new task via inference alone by conditioning on a few input-label pairs (demonstrations) and making predictions for new inputs. However, there has been little understanding of how the model learns and which aspects of the demonstrations contribute to end task performance. In this paper, we show that ground truth demonstrations are in fact not required -- randomly replacing labels in the demonstrations barely hurts performance on a range of classification and multi-choce tasks,…
saved by
related reading
- rethinking the role of demonstrations: what makes in-context learning work?arxiv.org
- How does in-context learning work? A framework for understanding the differences from traditional supervised learning | SAIL Blogai.stanford.edu
- In-context Learning and Induction Headstransformer-circuits.pub
- Learning without training: The implicit dynamics of in-context learningarxiv.org
- [2309.01809] Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
- Recursive Language Models | Alex L. Zhangalexzhang13.github.io
- In-context learning creates task vectorsarxiv.org
- Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
- [2005.14165] Language Models are Few-Shot Learnersarxiv.org
- Introducing S1: In-Context Learning for Roboticsskild.ai
- [2311.06668] In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringarxiv.org
- What Can Transformers Learn In-Context?arxiv.org