Mixture of experts - Wikipedia

Sorry to interrupt, but today's fundraiser won't last long This Wednesday, we ask you to join the 2% of readers who give. If everyone reading this right now gave just $3, we'd hit our goal in a couple of hours. $3 is all we ask. GIVE $3 MAYBE LATER Proud hosts of Wikipedia and its sister sites Mixture of experts (MoE) is a machine learning technique where multiple expert networks (learners) are used to divide a problem space into homogeneous regions.[1] It differs from ensemble techniques in that typically only one or a few expert models will be run, rather than combining results from all models. In mixture of experts, we always have the following ingredients, but they are constructed and combined differently. Both the experts and the weighting function are trained by minimizing some form of loss function, generally by gradient descent. There is a lot of freedom in choosing the precise form of experts, the weighting function, and the loss function. The meta-pi network, reported by Hamp

Mixture of experts - Wikipedia Jump to content From Wikipedia, the free encyclopedia Machine learning technique "MoE" redirects here. For other uses, see Moe . Part of a series on Machine learning and data mining Paradigms Supervised learning Unsupervised learning Semi-supervised learning Self-supervised learning Reinforcement learning Meta-learning Online learning Batch learning Curriculum learning Rule-based learning Neuro-symbolic AI Neuromorphic engineering Quantum machine learning Problems Classification Generative modeling Regression Clustering Dimensionality reduction Density estimation

related reading

• Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
• Mixture of Experts Explainedhuggingface.co
• Very Simple MoE Intro1a3orn.com
• Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scaleopenathena.ai
• [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
• Mixtral of Expertsarxiv.org
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
• [2209.01667] A Review of Sparse Expert Models in Deep Learningarxiv.org
• Monet: Mixture of Monosemantic Experts for Transformers Explained — LessWronglesswrong.com
• [2202.08906] ST-MoE: Designing Stable and Transferable Sparse Expert Modelsar5iv.labs.arxiv.org
• MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai