The economics of speculative decoding | Doubleword
Two underexplored axes: what MoE routing does to the decode roofline, and how compressed attention takes away the slack that used to make speculated tokens free.
Speculative decoding is one of the cleanest performance wins in inference optimisation: it's lossless, it hits decode latency when not much else does, and in its standard formulation it's simple and elegant. It works by looking forwards: speculative decoding takes a position on what tokens will come next. For dense transformers the bet is riskless: accepted tokens pay off, rejected tokens cost nothing, a clean arbitrage on spare memory bandwidth. A burst of research activity has recently pushed the envelope on how far forwards we can take that bet, for example Eagle 3.1 , DFlash , SSD . This p
saved by
related reading
- Speculative Decoding - philkravphilkrav.com
- Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com
- Better MoE model inference with warp decode · Cursorcursor.com
- Looking back at speculative decodingresearch.google
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- Speculative decodingaarnphm.xyz
- Fast Inference from Transformers via Speculative Decodingarxiv.org
- [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org