More Efficient In-Context Learning with GLaM – Google Research Blog

Large language models (e.g., GPT-3) have many significant capabilities, such as performing few-shot learning across a wide array of tasks, including reading comprehension and question answering with very few or no training examples. While these models can perform better by simply using more parameters, training and serving these large models can be very computationally intensive. Is it possible to train and use these models more efficiently? In “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts”, we introduce the Generalist Language Model (GLaM), a trillion weight model that can be trained and served efficiently (in terms of computation and energy use) thanks to sparsity, and achieves competitive performance on multiple few-shot learning tasks. GLaM’s performance compares favorably to a dense language model, GPT-3 (175B) with significantly improved learning efficiency across 29 public NLP benchmarks in seven categories, spanning language completion, open-domain questio

More Efficient In-Context Learning with GLaM Skip to main content More Efficient In-Context Learning with GLaM December 9, 2021 Posted by Andrew M Dai and Nan Du, Research Scientists, Google Research, Brain Team Quick links Share Copy link × Large language models (e.g., GPT-3 ) have many significant capabilities, such as performing few-shot learning across a wide array of tasks, including reading comprehension and question answering with very few or no training examples. While these models can perform better by simply using more parameters, training and serving these large models can be very c

saved by

• Braeden Hall

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• The Scaling Hypothesis · Gwern.netgwern.net
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• Review for NeurIPS paper: Language Models are Few-Shot Learnersproceedings.neurips.cc
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• LLM Resourcesforrestbicker.com
• Large Language Models Reading List | Sebastian Raschka, PhDsebastianraschka.com
• [2202.08906] ST-MoE: Designing Stable and Transferable Sparse Expert Modelsar5iv.labs.arxiv.org
• [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
• GLaM: Efficient Scaling of Language Models with Mixture-of-Expertsarxiv.org
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Mosaic LLMs: GPT-3 quality formosaicml.com