flâneur

Mixture-of-Kittens: our open-source MoE megakernel for NVL72s · Cursor

cursor.com · 4,694 words · saved by 3 readers

We're open-sourcing Mixture-of-Kittens, a deterministic MoE training megakernel for NVL72s that fuses communication and computation into a single kernel.

Today, we're open-sourcing Mixture-of-Kittens (MoK), our production MoE training megakernel for NVL72s. As we have scaled the training and inference of Composer, our agentic coding model, the mixture-of-experts layer has consistently remained the major bottleneck. Depending on the workload and training configuration, it can consume more than half of end-to-end training time. MoK addresses that bottleneck by fusing all MoE communication and computation into a single, fully deterministic kernel. It now powers Composer training across tens of thousands of GPUs. You can try MoK and explore…

saved by

related reading