GLaM: Efficient Scaling of Language Models with Mixture-of-Experts

N/A

GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du * 1 Yanping Huang * 1 Andrew M. Dai * 1 Simon Tong 1 Dmitry Lepikhin 1 Yuanzhong Xu 1 Maxim Krikun 1 Yanqi Zhou 1 Adams Wei Yu 1 Orhan Firat 1 Barret Zoph 1 Liam Fedus 1 Maarten Bosma 1 Zongwei Zhou 1 Tao Wang 1 Yu Emma Wang 1 Kellie Webster 1 Marie Pellat 1 Kevin Robinson 1 Kathleen Meier-Hellstern 1 Toju Duke 1 Lucas Dixon 1 Kun Zhang 1 Quoc V…

related reading

• More Efficient In-Context Learning with GLaMblog.research.google
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• The Scaling Hypothesis · Gwern.netgwern.net
• Pathways Language Model (PaLM): Scaling to 540 Billion Parameters for Breakthrouai.googleblog.com
• Mixtral of Expertsarxiv.org
• [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
• 2005.14165arxiv.org
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
• GLM-5.3-Flash: Frontier Intelligence, Flash Costz.ai
• Mosaic LLMs: GPT-3 quality formosaicml.com
• 2403.09611.pdfarxiv.org