Awesome-ML-SYS-Tutorial/sglang/scheduler/readme-en.md at main · zhaochenyang20/Awesome-ML-SYS-Tutorial

My learning notes for ML SYS. Contribute to zhaochenyang20/Awesome-ML-SYS-Tutorial development by creating an account on GitHub.

From KV Cache to Zero Overhead Scheduling: Understanding SGLang's Scheduling Ingenuity I've been wanting to systematically study the SGLang Scheduler for a year now. Around November '24, some friends and I wrote the SGLang Code Walkthrough , but we stopped at KV Cache Management and Scheduler analysis. Since then, a whole year has passed, and the SGLang project has undergone earth-shaking changes. However, in my opinion, the scheduling algorithm itself has long since converged. The most significant innovation remains the zero-overhead batch scheduler completed by lianmin in December '24. Since

saved by

• Chi McIsaac

related reading

• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• AgentIR: Workflow Aware Predictive Scheduling for Agent Graph LLM Serving - Krish Modi | Krish Modikrishmodi.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• Full-Stack Optimizations for Agentic Inference | NVIDIA Dynamo Documentationdocs.nvidia.com
• Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programsarxiv.org
• LLM Engineer's Almanac - Workloads | Modalmodal.com
• KV Caching Explained: Optimizing Transformer Inference Efficiencyhuggingface.co