✳flâneur — a map of the web's best reading

[2604.01193] Embarrassingly Simple Self-Distillation Improves Code Generation

arxiv.org · 654 words · saved by 1 readers

Abstract:Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation.

[2604.01193] Embarrassingly Simple Self-Distillation Improves Code Generation Skip to main content arXiv is now an independent nonprofit! Learn more × Search arXiv Press Enter to search · Advanced search --> Computer Science > Computation and Language arXiv:2604.01193 (cs) [Submitted on 1 Apr 2026 ( v1 ), last revised 24 Jun 2026 (this version, v2)] Title: Embarrassingly Simple Self-Distillation Improves Code Generation Authors: Ruixiang Zhang , Richard He Bai , Huangjie Zheng , Navdeep Jaitly , Ronan Collobert , Yizhe Zhang View a PDF of the paper titled Embarrassingly Simple Sel

Explore this link on the map →

related reading