Inside vLLM: Anatomy of a High-Throughput LLM Inference System | vLLM Blog
How vLLM's inference engine works, covering PagedAttention, continuous batching, prefix caching, speculative decoding, multi-GPU serving, scheduling, and benchm
Note: Originally posted on Aleksa Gordic's website. From paged attention, continuous batching, prefix caching, specdec, etc. to multi-GPU, multi-node dynamic serving at scale In this post, I'll gradually introduce all of the core system components and advanced features that make up a modern high-throughput LLM inference system. In particular I'll be doing a breakdown of how vLLM [1] works. This post is the first in a series. It starts broad and then layers in detail (following an inverse-pyramid approach) so you can form an accurate high-level mental model of the complete system without…
saved by
related reading
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- Introduction to vLLM and PagedAttentionblog.runpod.io
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- How LLM Inference Worksarpitbhayani.me
- Optimizing inference · Hugging Facehuggingface.co
- LLM Engineer's Almanac - Workloads | Modalmodal.com