How Kimi K3 Pushes MoE to the Extreme Without Breaking - YouTube
Kimi K3 shocked the tech world: on coding, knowledge, and reasoning benchmarks, it goes toe-to-toe with the strongest closed-source models, at 2.8 trillion p...
Kimi K3 shocked the tech world: on coding, knowledge, and reasoning benchmarks, it goes toe-to-toe with the strongest closed-source models, at 2.8 trillion parameters, with only 104 billion active per token. This video explains the architecture behind that gap: Stable LatentMoE. We build it from first principles: Latent MoE, which makes 896 experts affordable, and the three mechanisms that keep training stable at 16-of-896 sparsity: RMSNorm on the routed branch, SiTU-GLU inside each expert, and Quantile Balancing on the router. Chapters: 0:00 Kimi K3: 2.8T open-weight model 01:10 FFN/MoE:…
saved by
related reading
- How trillion-parameter AI models work and how to serve themyoutube.com
- ali (@waterloo_intern) on Xx.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
- Quantization from the ground upngrok.com
- Kimi K3 Tech Blog: Open Frontier Intelligencekimi.com
- Frontier-scale RL with Kimi K3appliedcompute.com
- Kimi Linear: An Expressive, Efficient Attention Architecturearxiv.org
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Kimi Linear: An Expressive, Efficient Attention Architecturealphaxiv.org
- Meta harness makes 10 times better Kimi K3 chipluoluo.ai
- Kimi k1.5: Scaling Reinforcement Learning with LLMsalphaxiv.org