Compute Allocation for AI Discovery and Search - Dmitry Rybin
An essay on Search and Discovery with AI Systems, arguing that compute allocation should consider three explicit components: training, inference, and exact algorithms.
TL;DR: I argue that AI systems for search and discovery should be designed around three compute budgets: \(C_{train}\), \(C_{infer}\), and \(C_{exact}\). Inference-Time Compute Discussion of inference-time compute has become mainstream since the rise of reasoning LLMs (o1, R1, o3). However, inference-time compute scaling is a classical idea that dates back to early AI systems. It is explicitly mentioned in: Claude Shannon's 1950 essay "Programming a Computer for Playing Chess"1; Monte Carlo Tree Search in AlphaGo by DeepMind in 20162; Inference-time search for Poker Bots, by Noam Brown…
saved by
related reading
- Spending Inference Time - Kevin Lukevinlu.ai
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- As Rocks May Think | Eric Jangevjang.com
- Optimally allocating compute between inference and training | Epoch AIepochai.org
- Optimally allocating compute between inference and trainingepoch.ai
- OpenAI’s Strawberry and inference scaling lawsinterconnects.ai
- Trading off compute in training and inference | Epoch AIepochai.org
- My picture of the present in AI — LessWronglesswrong.com
- Navigating the High Cost of AI Compute | Andreessen Horowitza16z.com
- o3 — LessWronglesswrong.com
- AI progress is about to speed up | Epoch AIepoch.ai
- Distinguish between inference scaling and "larger tasks use more compute" — AI Alignment Forumalignmentforum.org