flâneur

Better MoE model inference with warp decode · Cursor

cursor.com · 1,563 words · saved by 7 readers

By flipping the parallelism axis we achieve 1.8x faster and more accurate MoE model inference.

Blog / research Most MoE inference systems organize the token generation path around experts. This mirrors how routing works and has been the standard approach at scale. For small-batch decode on Blackwell GPUs, however, we found that organizing the kernel around outputs rather than experts works better. We call this approach “warp decode.” We arrived at warp decode by thinking about what the maximum achievable memory bandwidth for MoE decode on Blackwell actually is. That led us to flip the parallelism axis entirely. Instead of assigning warps to experts, we assign each warp to a single outpu

saved by

related reading