flâneur

Optimally Allocating Compute Between Inference and Training – Epoch

epochai.org · 2,315 words · saved by 1 readers

If it is feasible to trade off inference and training compute, we find that it is optimal for AI labs to spend similar amounts on training and inference. Sam Altman recently claimed that OpenAI currently generates around 100 billion tokens per day, or about 36 trillion tokens per year. Given that modern language models are trained on the order of 10 trillion tokens and tokens seen during training are around three times more expensive1 compared to tokens seen or generated during inference, a naive analysis2 suggests OpenAI’s annual inference costs are on the same order as their annual model training costs. This seems like an odd coincidence at first: why should one of these not completely dominate the other? However, there’s a good reason to suppose that these two quantities should be on a similar order of magnitude: the training-inference compute tradeoff. In this post, I will briefly explain what this tradeoff is about and why the current empirical evidence about it implies we should

Optimally allocating compute between inference and training | Epoch AI Introduction Sam Altman recently claimed that OpenAI currently generates around 100 billion tokens per day, or about 36 trillion tokens per year. Given that modern language models are trained on the order of 10 trillion tokens and tokens seen during training are around three times more expensive 1 compared to tokens seen or generated during inference, a naive analysis 2 suggests OpenAI’s annual inference costs are on the same order as their annual model training costs. This seems like an odd coincidence at first: why should

saved by

related reading