Speculative Decoding - philkrav
Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023, two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I’ll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one.
Speculative Decoding January 5, 2024 Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023 , two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I'll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one. Each new token requires a forward pass
saved by
related reading
- Looking back at speculative decodingresearch.google
- Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Speculative decodingaarnphm.xyz
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- The economics of speculative decoding | Doublewordblog.doubleword.ai
- [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org
- Speculative Speculative Decodingarxiv.org