Continuous batching from first principles
huggingface.co · 3,237 words · saved by 1 readers
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput. If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency. That's because at the heart of it, all LLMs are just fancy next token predictors. An LLM first processes your entire prompt to produce one new token. Then it keeps adding tokens one by one, each time…
saved by
related reading
- Inference economics of language modelsarxiv.org
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
- How to make LLMs go fastvgel.me
- How LLM Inference Worksarpitbhayani.me
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- How LLMs Actually Work | 0xkato0xkato.xyz
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
- KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co