[2005.14165] Language Models are Few-Shot Learners

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

[2005.14165] Language Models are Few-Shot Learners Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computation and Language arXiv:2005.14165 (cs) [Submitted on 28 May 2020 ( v1 ), last revised 22 Jul 2020 (this version, v4)] Title: Language Models are Few-Shot Learners Authors: Tom B. Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gr

related reading

• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Review for NeurIPS paper: Language Models are Few-Shot Learnersproceedings.neurips.cc
• Recent Advances in Language Model Fine-tuningruder.io
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• Few-Shot Prompting | Prompt Engineering Guidepromptingguide.ai
• More Efficient In-Context Learning with GLaMblog.research.google
• Self-Adapting Language Modelsarxiv.org
• 2403.09611.pdfarxiv.org
• Language Models can Solve Computer Tasksarxiv.org