Limits to narrow LLM complementarity @ osmarks' website

Taste is probably not the bottleneck.

Since late 2024, frontier LLM systems have been trained with high-compute reinforcement learning on outcome rewards[1], as opposed to the previous dominance of reinforcement learning from human feedback (smaller-scale due to the need for labelled data) and self-supervised pretraining. Broadly, this makes them better at "doing tasks" much more quickly and cheaply[2] than scaling pretraining would, but scale improves everything and this is not so general. This has led to some predictions along the lines of "taste[3] is the bottleneck". I don't believe that this has any more long-term validity…

saved by

• Timothy Kostolansky

related reading

• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• LLMs reward expertiseseangoedecke.com
• Will scaling work?dwarkeshpatel.com
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• llm assistant personas seem increasingly incoherent (some subjective observations) — LessWronglesswrong.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Human Data is (Probably) More Expensive Than Compute for Training Frontier LLMsddkang.substack.com
• Thoughts on AI in academiatheinfinitesimal.substack.com
• Thoughts on AI progress (Dec 2025)substack.com
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Simulated Users & Sad LLMs1a3orn.com
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org