Spending Inference Time — Kevin Lu

Inference is king: at the end of the day, the user wants a mapping from the input query to their desired output. While the two big ideas from The Bitter Lesson are learning (~= training) and search (~= inference), at the end of the day, people don’t really care about how much the model cost to train. But they will measure how slow the model is to milliseconds and put up the API cost on big leaderboards. If you tell the user it will take 5s but they have to wait 10s, they will end up leaving and not coming back. So, how should we think about getting the most bang for our inference buck? Traditionally, search has been very powerful, leading to AI’s superhuman capabilities (DeepBlue, AlphaGo). Nowadays, the model’s forward pass alone can even be prohibitively expensive, and the open-source community frequently tries to fit models into consumer GPUs (eg, Stable Diffusion). I think inference is really interesting because: At a high level, inference typically has two key components – a model

Spending Inference Time How should we structure inference compute to maximize performance? Published March 2024 Inference is king: at the end of the day, the user wants a mapping from the input query to their desired output. While the two big ideas from The Bitter Lesson are learning (~= training) and search (~= inference), at the end of the day, people don’t really care about how much the model cost to train. But they will measure how slow the model is to milliseconds and put up the API cost on big leaderboards. If you tell the user it will take 5s but they have to wait 10s, they will end up

related reading

• Trading off compute in training and inference | Epoch AIepochai.org
• Optimally allocating compute between inference and training | Epoch AIepochai.org
• The State of LLM Reasoning Model Inferencesebastianraschka.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• Best practices to accelerate inference for large-scale production workloadstogether.ai
• Inference.net | Full-Stack LLM Lifecycle Platforminference.net
• Composer2.pdfcursor.com
• Distinguish between inference scaling and "larger tasks use more compute" — AI Alignment Forumalignmentforum.org
• o3 — LessWronglesswrong.com
• Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
• Optimizing inference · Hugging Facehuggingface.co
• AI Model & API Providers Analysis | Artificial Analysisartificialanalysis.ai