Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Next
An interactive tutorial on speculative decoding: why it is faster, why rejection sampling keeps it lossless, how to evaluate it, and what comes next. NeurIPS 2026 Education Track.
Affiliations Independent Independent Venue NeurIPS 2026 Education Track (under review) Introduction What it is and why it matters Every LLM you use generates one token at a time. This is , which is a major bottleneck of inference: producing n tokens takes n passes through a model with tens of billions of parameters. , introduced in 2023 (Leviathan et al., 2023), is an approach to accelerate this: a lightweight proposes the next few tokens, and the full verifies these drafts, accepting or rejecting each one. The verified output has the same as the target model (Chen et al., 2023), which…
saved by
related reading
- Speculative Decoding - philkravphilkrav.com
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org
- How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Looking back at speculative decodingresearch.google
- Speculative Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Two different tricks for fast LLM inferenceseangoedecke.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
- Speculative decodingaarnphm.xyz