LLM Engineer's Almanac - Advisor | Modal

A simple tool for estimating the throughput and latency of LLM engines

I want to serve with I expect on average Clients should receive in under 95% of the time I want to see configuration Metric: Aggregate: Frequently Asked Questions What is this? How do I use it? This interactive chart indicates the per-replica throughput and client-side latency you can expect when running open weights language models on open source inference engines, in particular on Modal. Select a workload (model, tokens in and out), set a latency objective, and indicate whether you want to see all configurations or just the one that got the best throughput or best latency.…

saved by

• Ratan Kaliani

related reading

• KernelBench: Can LLMs Write GPU Kernels?scalingintelligence.stanford.edu
• LLM Engineer's Almanac - Workloads | Modalmodal.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• LLM Architecture Gallery | Sebastian Raschka, PhDsebastianraschka.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• LLM Engineer's Almanac - Executive Summary | Modalmodal.com
• Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
• Patterns for Building LLM-based Systems & Productseugeneyan.com
• The State of LLM Serving in 2026: Ollama, SGLang, TensorRT, Triton, and vLLM | Canteenthecanteenapp.com
• LLM Inference Handbookhandbook.modular.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• Inside vLLM: Anatomy of a High-Throughput LLM Inference Systemvllm.ai