Nemotron 3 Ultra: what distillation can't fix

Ten specialist teachers distilled into one open 550B model

Nemotron 3 Ultra: what distillation can't fix Ten specialist teachers distilled into one open 550B model Maxime Labonne Jun 08, 2026 19 3 2 Share On June 4, 2026, Nvidia closed out its Nemotron 3 lineup with Nemotron 3 Ultra , a 550B-parameter MoE with 55B active and fully open weights. It is the most intelligent open-weight model from a US lab, but it still trails the Chinese open frontier led by Kimi K2.6. Nvidia's pitch is not the highest benchmark scores, but the highest throughput on long-running agentic workloads . Let's walk through the recipe and poke at the inference numbers behind th

saved by

• Christina Lee

related reading

• [2605.23857] Strong Teacher Not Needed? On Distillation in LLM Pretrainingarxiv.org
• [2604.13010] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationarxiv.org
• nemotron-3-super-120b-a12b Model by NVIDIA | NVIDIA NIMbuild.nvidia.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• Inside Nemotron & NVIDIA’s AI Lab | Bryan Catanzaroyoutube.com
• >10x More Efficient Pretraining — Magicmagic.dev
• Nemotron_Diffusion_Tech_Report_v1.pdfd1qx31qr3h6wln.cloudfront.net
• Nitrobrew: Fast, Lossless Distillation for Free | Tildeblog.tilderesearch.com
• GitHub - NVIDIA-NeMo/Automodel: 🚀 Pytorch Distributed native training library for LLMs/VLMs with OOTB Hugging Face supportgithub.com
• Kimi K3 Tech Blog: Open Frontier Intelligencekimi.com
• Inside the Inference Hardware Revolution Of 2026spectrum.ieee.org