MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators

N/A

M I L O : E FFICIENT Q UANTIZED M O E I NFERENCE WITH M IXTURE OF L OW-R ANK C OMPENSATORS Beichen Huang * 1 2 Yueming Yuan * 1 Zelei Shao * 1 Minjia Zhang 1 A BSTRACT A critical approach for efficiently deploying Mixture-of-Experts (MoE) models with massive parameters is quantization. However, state-of-the-art MoE models suffer from non-negligible accuracy loss with extreme quantization, such as under 4 bits. To address this, we introduce MiLo, a…

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
• 2305.14314arxiv.org
• AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Accelerationarxiv.org
• How We Build Trillion Parameter Reasoning RL with 10% GPUsmacaron.im
• 2502.11089arxiv.org
• 2106.09685arxiv.org
• 2408.14690arxiv.org
• Mixture of Experts Quantile Balancing: Validated at 32B-A5B (1e22 FLOPs) Scaleopenathena.ai
• 2112.10769arxiv.org