Hamel’s Blog - Optimizing LLM latency

An exploration of inference tools for open source LLMs focused on latency.

Summary Below is a summary of my findings: 🏁 mlc is the fastest. This is so fast that I’m skeptical and am now motivated to measure quality (if I have time). When checking the outputs manually, they didn’t seem that different than other approaches. ❤️ CTranslate2 is my favorite tool, which is among the fastest but is also the easiest to use. The documentation is the best out of all of the solutions I tried. Furthermore, I think that the ergonomics are excellent for the models that they support. Unlike vLLM, CTranslate doesn’t seem to support distributed inference just yet. 🛠️ vLLM is…

related reading

• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Optimizing inference · Hugging Facehuggingface.co
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• Optimized Inference Deployment · Hugging Facehuggingface.co
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• LLM Engineer's Almanac - Advisormodal.com
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• Inference characteristics of Llama · Cursorcursor.com
• How is LLaMa.cpp possible?finbarr.ca
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com