✳flâneur — a map of the web's best reading

My AGI safety research—2025 review, ’26 plans — AI Alignment Forum

alignmentforum.org · 4,349 words · saved by 1 readers

“Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” –attributed to DL Moody[1] The main threat model I’m working to address is the same as it’s been since I was hobby-blogging about AGI safety in 2019. Basically, I think that: I think that, when this learning algorithm is understood, it will be easy to get it to do powerful and impressive things, and to make money, as long as it’s weak enough that humans can keep it under control. But past that stage, we’ll be relying on the AGIs to have good motivations, and not be egregiously misaligned and scheming to take over the world and wipe out humanity. Alas, I claim that the latter kind of motivation is what we should expect to occur, in the absence of yet-to-be-invented techniques to avoid it. Inventing those yet-to-be-invented techniques constitutes the technical alignment problem for brain-like AGI. That’s the main thing I’ve been working on since I’ve been in the field. See my Intro to

x My AGI safety research—2025 review, ’26 plans — AI Alignment Forum Research Agendas AI Frontpage 2025 Top Fifty: 9 % 47 My AGI safety research—2025 review, ’26 plans by Steven Byrnes 11th Dec 2025 15 min read 4 47 Previous: 2024 , 2022 “Our greatest fear should not be of failure, but of succeeding at something that doesn't really matter.” – attributed to DL Moody [1] 1. Background & threat model The main threat model I’m working to address is the same as it’s been since I was hobby-blogging about AGI safety in 2019. Basically, I think that: The “secret sauce” of human intelligence is a big u

Explore this link on the map →

related reading