Awesome-ML-SYS-Tutorial/sglang/scheduler/readme-en.md at main · zhaochenyang20/Awesome-ML-SYS-Tutorial

My learning notes for ML SYS. Contribute to zhaochenyang20/Awesome-ML-SYS-Tutorial development by creating an account on GitHub.

From KV Cache to Zero Overhead Scheduling: Understanding SGLang's Scheduling Ingenuity I've been wanting to systematically study the SGLang Scheduler for a year now. Around November '24, some friends and I wrote the SGLang Code Walkthrough , but we stopped at KV Cache Management and Scheduler analysis. Since then, a whole year has passed, and the SGLang project has undergone earth-shaking changes. However, in my opinion, the scheduling algorithm itself has long since converged. The most significant innovation remains the zero-overhead batch scheduler completed by lianmin in December '24. Since

saved by

• Chi McIsaac

related reading

• Lil'Loglilianweng.github.io
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
• Parallel CPU-GPU Execution for LLM Inference on Constrained GPUsarxiv.org
• Updating 1T parameters in seconds — P2P weight transfer in Large Scale Distributed RL - LMSYS Orglmsys.org
• Machine Learning System Resources | std::bodun::blogbodunhu.com