Better MoE model inference with warp decode · Cursor

By flipping the parallelism axis we achieve 1.8x faster and more accurate MoE model inference.

Blog / research Most MoE inference systems organize the token generation path around experts. This mirrors how routing works and has been the standard approach at scale. For small-batch decode on Blackwell GPUs, however, we found that organizing the kernel around outputs rather than experts works better. We call this approach “warp decode.” We arrived at warp decode by thinking about what the maximum achievable memory bandwidth for MoE decode on Blackwell actually is. That led us to flip the parallelism axis entirely. Instead of assigning warps to experts, we assign each warp to a single outpu

saved by

• Thu Than
• Christina Lee
• Arya G

related reading

• Mixture-of-Kittens: our open-source MoE megakernel for NVL72scursor.com
• RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
• DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0mlcommons.org
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Real-time LLM Inference on Standard Datacenter GPUs (3,000 tokens/s per request)blog.kog.ai
• The economics of speculative decoding | Doublewordblog.doubleword.ai
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• 1.5x faster MoE training with custom MXFP8 kernels · Cursorcursor.com
• Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Speculative Decoding - philkravphilkrav.com