✳flâneur — a map of the web's best reading

Nitrobrew: Fast, Lossless Distillation for Free | Tilde

blog.tilderesearch.com · 10,856 words · saved by 2 readers

Note: Shortly before this post went live, DeepSeek-V4 independently reported the same core idea for full-vocabulary OPD — caching teacher hidden states and reconstructing logits on the fly. Nitrobrew was developed independently and differs in several respects: open-source framework integrations (NeMo RL, VeRL), explicit online divergence algorithms for forward/reverse KL and JSD, detailed profiling across model scales, and spectral compression experiments (SVD-Nitrobrew) for further reducing communication below 𝑑 model d model ​ . In distillation, a student model is trained to reproduce the behaviour of a more capable teacher model. The teacher learns a compressed representation of the data distribution, which is then transferred to the student through supervision on soft labels, logits, or hidden states [1] [2]. Soft teacher outputs often provide richer supervision than hard labels because they expose the teacher's learned approximation to the noisy data-generating process, includ

Back Nitrobrew: Fast, Lossless Distillation for Free 4.28.2026 Dhruv Pai*, Timor Averbuch*, Alec Dewulf*, Ben Keigwin, Ashley Zhang * Core Contributor; Correspondence to dhruv@tilderesearch.com TL;DR Distillation, especially on-policy distillation, has become a crucial component in reasoning model post-training workflows. Logit distillation at modern vocabulary sizes is bottlenecked by communication and memory, not compute. Nitrobrew exploits the fact that teacher logits are generated from a much lower-dimensional hidden state through the unembedding matrix. Sends hidden states instead of logi

Explore this link on the map →

saved by

related reading