RL & search is a terrifying way to build AGI (an FAQ) — LessWrong
Q1: WHAT ARE YOU SAYING? A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via RL and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. …
x RL & search is a terrifying way to build AGI (an FAQ) — LessWrong Reinforcement learning Shard Theory AI Frontpage 2026 Top Fifty: 11 % 118 RL & search is a terrifying way to build AGI (an FAQ) by Steven Byrnes 27th Jul 2026 AI Alignment Forum 17 min read 13 118 Ω 38 Q1: What are you saying? A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. You’re playing aro
Explore this link on the map →related reading
- Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
- AGI Ruin: A List of Lethalities — LessWronglesswrong.com
- Alignment remains a hard, unsolved problem — LessWronglesswrong.com
- Proposal: Using Monte Carlo tree search instead of RLHF for alignment research — LessWronglesswrong.com
- Reward Function Design: a starter pack — LessWronglesswrong.com
- How To Go From Interpretability To Alignment: Just Retarget The Search — LessWronglesswrong.com
- Reward Is Not Enough — LessWronglesswrong.com
- Many arguments for AI x-risk are wrong — LessWronglesswrong.com
- Research Areas in Evaluation and Guarantees in Reinforcement Learning (The Alignment Project by UK AISI) — AI Alignment Forumalignmentforum.org
- Commentary on AGI Safety from First Principles — AI Alignment Forumalignmentforum.org
- “Behaviorist” RL reward functions lead to scheming — AI Alignment Forumalignmentforum.org
- My AGI safety research—2025 review, ’26 plans — AI Alignment Forumalignmentforum.org