The economics of speculative decoding | Doubleword
Two underexplored axes: what MoE routing does to the decode roofline, and how compressed attention takes away the slack that used to make speculated tokens free.
Speculative decoding is one of the cleanest performance wins in inference optimisation: it's lossless, it hits decode latency when not much else does, and in its standard formulation it's simple and elegant. It works by looking forwards: speculative decoding takes a position on what tokens will come next. For dense transformers the bet is riskless: accepted tokens pay off, rejected tokens cost nothing, a clean arbitrage on spare memory bandwidth. A burst of research activity has recently pushed the envelope on how far forwards we can take that bet, for example Eagle 3.1 , DFlash , SSD . This p
Explore this link on the map →saved by
related reading
- Speculative Decoding - philkravphilkrav.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Speculative Decoding - Deep Dive — ROCm Blogsrocm.blogs.amd.com
- Better MoE model inference with warp decode · Cursorcursor.com
- Speculative decodingaarnphm.xyz
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Best practices to accelerate inference for large-scale production workloadstogether.ai
- Xiaomi MiMo, Explore and Lovemimo.xiaomi.com
- 2502.11089arxiv.org
- The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev