How speculative decoding makes LLMs go brrr – Leonie Monigatti
leoniemonigatti.com · 3,464 words · saved by 1 readers
Learn how speculative decoding accelerates LLM inference without quality loss: the draft-and-verify algorithm, plus Medusa, EAGLE, DFlash, and DSpark.
Transformer-based Large Language Models (LLMs) generate text autoregressively. This means that text is generated token by token, where each token requires a full forward pass of the model. This sequential dependency makes inference latency proportional to output length, and thus slow. For latency-critical applications, such as real-time conversation or multi-turn agentic workflows, this is a real bottleneck in production speculative decoding aims to overcome. Speculative decoding is an inference optimization technique that reduces decoding latency while preserving output quality. It was…
saved by
related reading
- Speculative Decoding - philkravphilkrav.com
- Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
- Looking back at speculative decodingresearch.google
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Speculative Speculative Decodingarxiv.org
- [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Speculative Decoding - Deep Dive — ROCm Blogsrocm.blogs.amd.com
- Speculative decodingaarnphm.xyz
- DFlash: Block Diffusion for Flash Speculative Decoding - Z Labz-lab.ai