flâneur

MiLo: Efficient Quantized MoE Inference with Mixture of Low-Rank Compensators

beichenhuang.github.io · 5,605 words · saved by 1 readers

N/A

M I L O : E FFICIENT Q UANTIZED M O E I NFERENCE WITH M IXTURE OF L OW-R ANK C OMPENSATORS Beichen Huang * 1 2 Yueming Yuan * 1 Zelei Shao * 1 Minjia Zhang 1 A BSTRACT A critical approach for efficiently deploying Mixture-of-Experts (MoE) models with massive parameters is quantization. However, state-of-the-art MoE models suffer from non-negligible accuracy loss with extreme quantization, such as under 4 bits. To address this, we introduce MiLo, a…

related reading