[2506.04645] Inference economics of language models

Abstract:We develop a theoretical model that addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale. Our model takes into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes to find the ones that optimize serial inference speed at a given cost per token. We use the model to compute Pareto frontiers of serial speed versus cost per token for popular language models.

View PDF HTML (experimental) Abstract:We develop a theoretical model that addresses the economic trade-off between cost per token versus serial token generation speed when deploying LLMs for inference at scale. Our model takes into account arithmetic, memory bandwidth, network bandwidth and latency constraints; and optimizes over different parallelism setups and batch sizes to find the ones that optimize serial inference speed at a given cost per token. We use the model to compute Pareto frontiers of serial speed versus cost per token for popular language models. Subjects: Machine…

saved by

• Arav Kumar

related reading

• Scalable MatMul-free Language Modelingarxiv.org
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• LLM Inference Economics from First Principlestensoreconomics.com
• [2509.14786] Pre-training under infinite computearxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• Fast Inference from Transformers via Speculative Decodingarxiv.org
• [2605.06546] Efficient Pre-Training with Token Superpositionarxiv.org
• [2604.22750] How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasksarxiv.org
• [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
• Efficiently Scaling Transformer Inferencearxiv.org