Distillation Walkthrough
vladfeinberg.com · 1,446 words · saved by 2 readers
Vlad's Blog
Distillation Walkthrough Distillation is a critical technique towards improving a network’s quality while keeping its serving latency constant. This is becoming crucial as people focus on serving larger and larger models. Image found on LinkedIn . Distillation is a powerful technique, but a couple things about it are quite mystical. The purpose of this post is to: Provide a very high level explainer (but mostly refer to source papers) of distillation. Show that you can create a simple, linear example where distillation works. How is distillation implemented? At an algorithmic level, distillati
saved by
related reading
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- What is Model Distillation?labelbox.com
- [1511.03643] Unifying distillation and privileged informationarxiv.org
- Nitrobrew: Fast, Lossless Distillation for Free | Tildeblog.tilderesearch.com
- A Recipe for Training Neural Networkskarpathy.github.io
- Everything You Need to Know about Knowledge Distillationhuggingface.co
- Distill — Latest articles about machine learningdistill.pub
- Lil'Loglilianweng.github.io
- Bitter Lessons from Distillation Robustifies Unlearningbrucewlee.com
- Zero-Shot Knowledge Distillation in Deep Networksarxiv.org
- What are Diffusion Models? | Lil'Loglilianweng.github.io