Noam Brown on X: "Implications of Large-Scale Test-Time Compute" / X

https://t.co/oWqzT12RtZ

@polynoamial: Implications of Large-Scale Test-Time Compute tl;dr: As LLMs become more capable, benchmark performance is increasingly a function of test-time compute. In fact, we likely don't know what the capability ceiling is for modern LLMs because it's too expensive to measure. We should change LLM evaluations to account for that by measuring performance vs tokens, cost, or time. The day GPT-5.5 was released, the initial reaction was skepticism. The benchmark numbers were better, but not by much: However, within hours, once people had time to play around with the model, it became clear th

saved by

• Winnie Xu
• Information Flow
• Varun Goyal
• Ishan Mukherjee
• Devin Plumb
• Ishaan Panigrahi
• Tom Zheng

related reading

• Latency Scaling Differences for GPT and Claude Modelsepoch.ai
• On measuring AInikilravi.substack.com
• The nature of LLM algorithmic progress (v2) — LessWronglesswrong.com
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Humans Still Beat AI in the Long Horizon: Revisiting Test-Time Scaling in the Agent Era | Qiuyang Mangjoyemang33.github.io
• Pretraining progress is mostly coming from datadwarkesh.com
• Will scaling work?dwarkeshpatel.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• More compute, more capability: Why AI agent evaluations need to account for test-time compute | AISI Workaisi.gov.uk
• I. From GPT-4 to AGI: Counting the OOMs - SITUATIONAL AWARENESSsituational-awareness.ai
• AI in 2025: gestalt — LessWronglesswrong.com
• AI progress is about to speed up | Epoch AIepoch.ai