Speculative Decoding - philkrav
Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023, two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I’ll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one.
Speculative Decoding January 5, 2024 Speculative Decoding LLM inference will constitute an increasingly large proportion of compute cost. Unfortunately, for autoregressive LLMs, it is slow. Speculative decoding is a clever technique described by both Leviathan et al. 2022 and Chen et al. 2023 , two concurrent papers (somewhat amusingly, from Google Research and Deepmind respectively). I'll explain the technique, its derivation, and newer variants in this post. Autoregressive sampling is typically memory bandwidth bound since tokens are sampled one-by-one. Each new token requires a forward pass
Explore this link on the map →saved by
related reading
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Speculative decodingaarnphm.xyz
- Speculative Decoding - Deep Dive — ROCm Blogsrocm.blogs.amd.com
- The economics of speculative decoding | Doublewordblog.doubleword.ai
- DFlash: Block Diffusion for Flash Speculative Decoding - Z Labz-lab.ai
- Optimizing inference · Hugging Facehuggingface.co
- How to make LLMs go fastvgel.me
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- How LLMs Actually Work | 0xkato0xkato.xyz
- Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
- Composer2.pdfcursor.com