Decoding Speculative Decoding from First Principles / JW Labs

JW Labs research post.

Prereq knowledge on autoregressive language models and hardware Large language models work like this: Old tokens + KV is fed through the model to predict the new token and KV. This is done in an iterative loop to keep generating next tokens. This loop needs to run even when reading the user's query (though the predicted next token is discarded). Where token is the word outputted, and KV is the cumulative knowledge of the sentence that gets iteratively built up. You should observe that generation is sequential. Token #50 depends on token #49 which depends on token #48 and so forth. This…

saved by

• Benedict Neo

related reading

• Speculative Decoding: How It Evolved, When It Stays Lossless, and What's Nextneurips2026-speculative-decoding.vercel.app
• [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• Speculative Decoding - philkravphilkrav.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Two different tricks for fast LLM inferenceseangoedecke.com
• Looking back at speculative decodingresearch.google
• How speculative decoding makes LLMs go brrr – Leonie Monigattileoniemonigatti.com
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org