RL & search is a terrifying way to build AGI (an FAQ) — LessWrong
Q1: WHAT ARE YOU SAYING? A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via RL and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. …
x RL & search is a terrifying way to build AGI (an FAQ) — LessWrong Reinforcement learning Shard Theory AI Frontpage 2026 Top Fifty: 11 % 118 RL & search is a terrifying way to build AGI (an FAQ) by Steven Byrnes 27th Jul 2026 AI Alignment Forum 17 min read 13 118 Ω 38 Q1: What are you saying? A: My claim here is that if you build artificial general intelligence (AGI) via any algorithm that’s choosing actions via reinforcement learning (RL) and/or model-based search and planning—a giant chunk of your AI textbook—then that’s just an utterly terrifying thing that you’re doing. You’re playing aro
saved by
related reading
- Why Tool AIs Want to Be Agent AIs · Gwern.netgwern.net
- The Case Against AI Control Research — LessWronglesswrong.com
- Artificial General Intelligence and how (much) to worry about itstrangeloopcanon.com
- Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover — LessWronglesswrong.com
- I am worried about near-term non-LLM AI developments — LessWronglesswrong.com
- What failure looks like — AI Alignment Forumalignmentforum.org
- “Behaviorist” RL reward functions lead to scheming — AI Alignment Forumalignmentforum.org
- My AGI safety research—2025 review, ’26 plans — AI Alignment Forumalignmentforum.org
- AGI safetysjbyrnes.com
- Questions about the Future of AI - by Dwarkesh Pateldwarkesh.com
- Clarifying and predicting AGI — LessWronglesswrong.com
- Many arguments for AI x-risk are wrong — AI Alignment Forumalignmentforum.org