Papers I’ve read this week, Mixture of Experts edition

I read a bunch of papers about conditional routing models

Papers I’ve read this week, Mixture of Experts edition I read a bunch of papers about conditional routing models Finbarr Timbers Aug 04, 2023 40 3 1 Share Papers I’ve read this week, Mixture of Experts edition Mixture of Experts (MoE) models have been getting a lot of attention lately, what with the all the rumours about OpenAI using them in GPT-4. I’ve been reading a lot of the foundational papers about MoE models, and I’ve taken detailed notes, which I wanted to share. This is a bit of a long one, so you might want to read this on the web. Background A standard deep learning model uses the s

saved by

• Shubham Chandel
• Roshan Swaroop

related reading

• Linear Transformers Are Faster After All – Manifest AImanifestai.com
• Efficient LLM inferencefinbarrtimbers.substack.com
• Non_Interactive – Software & MLnonint.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Mixture of Experts Explainedhuggingface.co
• Very Simple MoE Intro1a3orn.com
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org
• [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
• [2209.01667] A Review of Sparse Expert Models in Deep Learningarxiv.org
• Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scaleopenathena.ai
• An Extremely Opinionated Annotated List of My Favourite Mechanistic Interpretability Papers v2 — AI Alignment Forumalignmentforum.org