Mixtral of Experts

N/A

Mixtral of Experts Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang,…

related reading

• Welcome Mixtral - a SOTA Mixture of Experts on Hugging Facehuggingface.co
• Papers I’ve read this week, Mixture of Experts editionfinbarrtimbers.substack.com
• Large Language Diffusion Modelsarxiv.org
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2605.12715] Scaling Laws for Mixture Pretraining Under Data Constraintsarxiv.org
• Esoteric Language Modelsarxiv.org
• Mixture of Experts Explainedhuggingface.co
• [2101.03961] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsityarxiv.org
• Papers Explained 64: Mistralmedium.com
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io