Mixtral of Experts
arxiv.org · 4,952 words · saved by 1 readers
N/A
Mixtral of Experts Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang,…
related reading
- Welcome Mixtral - a SOTA Mixture of Experts on Hugging Facehuggingface.co
- Composer2.pdfcursor.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
- Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
- GLaM: Efficient Scaling of Language Models with Mixture-of-Expertsarxiv.org
- Mixture of Experts Explainedhuggingface.co
- MAI-Thinking-1: Building a Hill-Climbing Machinemicrosoft.ai
- Mixture of experts - Wikipediaen.wikipedia.org
- [1701.06538] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layerarxiv.org
- More Efficient In-Context Learning with GLaMblog.research.google
- [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org