Awesome-ML-SYS-Tutorial/sglang/scheduler/readme-en.md at main · zhaochenyang20/Awesome-ML-SYS-Tutorial

My learning notes for ML SYS. Contribute to zhaochenyang20/Awesome-ML-SYS-Tutorial development by creating an account on GitHub.

From KV Cache to Zero Overhead Scheduling: Understanding SGLang's Scheduling Ingenuity I've been wanting to systematically study the SGLang Scheduler for a year now. Around November '24, some friends and I wrote the SGLang Code Walkthrough , but we stopped at KV Cache Management and Scheduler analysis. Since then, a whole year has passed, and the SGLang project has undergone earth-shaking changes. However, in my opinion, the scheduling algorithm itself has long since converged. The most significant innovation remains the zero-overhead batch scheduler completed by lianmin in December '24. Since

saved by

• Chi McIsaac

related reading

• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUsarxiv.org
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• Continuous batching from first principleshuggingface.co
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• AgentIR: Workflow Aware Predictive Scheduling for Agent Graph LLM Serving - Krish Modi | Krish Modikrishmodi.com
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
• Efficient Memory Management for Large Language Model Serving with PagedAttentionarxiv.org