Continuous batching from first principles
huggingface.co · 3,237 words · saved by 1 readers
We’re on a journey to advance and democratize artificial intelligence through open source and open science.
TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput. If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes a while for the first word of the response to appear, and then words appear one-by-one on your screen with (hopefully) a regular and fast-paced frequency. That's because at the heart of it, all LLMs are just fancy next token predictors. An LLM first processes your entire prompt to produce one new token. Then it keeps adding tokens one by one, each time…
saved by
related reading
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- All About Transformer Inferencejax-ml.github.io
- KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- How LLMs Actually Work | 0xkato0xkato.xyz
- Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- ali (@waterloo_intern) on Xx.com