GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

N/A

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du * 1 Yanping Huang * 1 Andrew M. Dai * 1 Simon Tong 1 Dmitry Lepikhin 1 Yuanzhong Xu 1 Maxim Krikun 1 Yanqi Zhou 1 Adams Wei Yu 1 Orhan Firat 1 Barret Zoph 1 Liam Fedus 1 Maarten Bosma 1 Zongwei Zhou 1 Tao Wang 1 Yu Emma Wang 1 Kellie Webster 1 Marie Pellat 1 Kevin Robinson 1 Kathleen Meier-Hellstern 1 Toju Duke 1 Lucas Dixon 1 Kun Zhang 1 Quoc V…

related reading

• More Efficient In-Context Learning with GLaMblog.research.google
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Large Language Diffusion Modelsarxiv.org
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• Language Model Compositionarxiv.org
• 2005.14165arxiv.org
• Generalized Language Models | Lil'Loglilianweng.github.io
• A Family of Large Language Models from GLM-130B to GLM-4 All Toolsarxiv.org
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org