Compute Allocation for AI Discovery and Search - Dmitry Rybin
An essay on Search and Discovery with AI Systems, arguing that compute allocation should consider three explicit components: training, inference, and exact algorithms.
TL;DR: I argue that AI systems for search and discovery should be designed around three compute budgets: \(C_{train}\), \(C_{infer}\), and \(C_{exact}\). Inference-Time Compute Discussion of inference-time compute has become mainstream since the rise of reasoning LLMs (o1, R1, o3). However, inference-time compute scaling is a classical idea that dates back to early AI systems. It is explicitly mentioned in: Claude Shannon's 1950 essay "Programming a Computer for Playing Chess"1; Monte Carlo Tree Search in AlphaGo by DeepMind in 20162; Inference-time search for Poker Bots, by Noam Brown…
saved by
related reading
- Spending Inference Time - Kevin Lukevinlu.ai
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- Optimally allocating compute between inference and trainingepoch.ai
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Navigating the High Cost of AI Compute | Andreessen Horowitza16z.com
- Optimally allocating compute between inference and training | Epoch AIepochai.org
- Trading off compute in training and inference | Epoch AIepochai.org
- Efficient LLM inferencefinbarrtimbers.substack.com
- Estimating training compute of deep learning models | Epoch AIepochai.org
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- [2412.14135] Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivearxiv.org