flâneur

Spending Inference Time — Kevin Lu

kevinlu.ai · 2,639 words · saved by 4 readers

Inference is king: at the end of the day, the user wants a mapping from the input query to their desired output. While the two big ideas from The Bitter Lesson are learning (~= training) and search (~= inference), at the end of the day, people don’t really care about how much the model cost to train. But they will measure how slow the model is to milliseconds and put up the API cost on big leaderboards. If you tell the user it will take 5s but they have to wait 10s, they will end up leaving and not coming back. So, how should we think about getting the most bang for our inference buck? Traditionally, search has been very powerful, leading to AI’s superhuman capabilities (DeepBlue, AlphaGo). Nowadays, the model’s forward pass alone can even be prohibitively expensive, and the open-source community frequently tries to fit models into consumer GPUs (eg, Stable Diffusion). I think inference is really interesting because: At a high level, inference typically has two key components – a model

Spending Inference Time How should we structure inference compute to maximize performance? Published March 2024 Inference is king: at the end of the day, the user wants a mapping from the input query to their desired output. While the two big ideas from The Bitter Lesson are learning (~= training) and search (~= inference), at the end of the day, people don’t really care about how much the model cost to train. But they will measure how slow the model is to milliseconds and put up the API cost on big leaderboards. If you tell the user it will take 5s but they have to wait 10s, they will end up

saved by

related reading