Speculative Decoding - philkrav
Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023, two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I’ll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one.
Speculative Decoding January 5, 2024 Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023 , two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I'll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one. Each new token requires a forward pass
saved by
related reading
- Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
- Looking back at speculative decodingresearch.google
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Speculative Speculative Decodingarxiv.org
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org
- Speculative Decoding - Deep Dive — ROCm Blogsrocm.blogs.amd.com
- Speculative decodingaarnphm.xyz
- 2302.01318arxiv.org