[2210.11416] Scaling Instruction-Finetuned Language Models

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we explore instruction finetuning with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) finetuning on chain-of-thought data. We find that instruction finetuning with the above aspects dramatically improves performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT), and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation). For instance, Flan-PaLM 540B instruction-finetuned on 1.8K tasks outperforms PALM 540B by a large margin (+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as 75.2% on five-shot MMLU. We also publicly release Flan-T5 checkpoints, which achieve strong few-shot performance even compared to much larger models, such as PaLM 62B. Overall, instruction finetuning is a general method for improving the performance and usability of pretrained language models.

[2210.11416] Scaling Instruction-Finetuned Language Models --> Computer Science > Machine Learning arXiv:2210.11416 (cs) [Submitted on 20 Oct 2022 ( v1 ), last revised 6 Dec 2022 (this version, v5)] Title: Scaling Instruction-Finetuned Language Models Authors: Hyung Won Chung , Le Hou , Shayne Longpre , Barret Zoph , Yi Tay , William Fedus , Yunxuan Li , Xuezhi Wang , Mostafa Dehghani , Siddhartha Brahma , Albert Webson , Shixiang Shane Gu , Zhuyun Dai , Mirac Suzgun , Xinyun Chen , Aakanksha Chowdhery , Alex Castro-Ros , Marie Pellat , Kevin Robinson , Dasha Valter , Sharan Narang , Gaurav Mi

saved by

• Fernando Silva

related reading

• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• Large Language Diffusion Modelsarxiv.org
• Recent Advances in Language Model Fine-tuningruder.io
• Stanford CRFMcrfm.stanford.edu
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• Difference between Instruction Tuning vs Non Instruction Tuning Large Language Models - Stack Overflowstackoverflow.com
• Language Models can Solve Computer Tasksarxiv.org
• Fine-tuning LLMs Guide | Unsloth Documentationdocs.unsloth.ai
• Modern Pretraining Strategies: A Hands-On Guidetheneuralmaze.substack.com
• LLM Resourcesforrestbicker.com