AGI and Lock-in | Forethought

The long-term future of intelligent life is currently unpredictable and undetermined. We argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competently pursue any such values for at least millions, and plausibly trillions, of years.

AGI and Lock-in Lukas Finnveden Jess Riedel Carl Shulman Authors Citations Cite Citations PDF Contact 8th October 2022 Last update: 11th March 2025 AGI and Lock-in Abstract 0 Summary 0.0 The claim 0.1 Preserving information 0.2 Executing intentions 0.3 Preventing disruption 0.4 Some things we don’t argue for 0.5 Structure of the document 1 To be more precise 1.1 What do we assume about “AGI”? 1.2 What do we mean by “lock-in”? 1.3 How confident are we that stability is feasible, for how long? 2 Desirability and probability 2.1 Would it be good for highly stable institutions to be built? 2.2 How

related reading

• Persistent Path-Dependence: Why Our Actions Matter Long-Termforethought.org
• AGI Ruin: A List of Lethalities — LessWronglesswrong.com
• Planning for AGI and beyond | OpenAIopenai.com
• What failure looks like — LessWronglesswrong.com
• Three Observations - Sam Altmanblog.samaltman.com
• What failure looks like — AI Alignment Forumalignmentforum.org
• Reframing AI Safety as a Neverending Institutional Challenge – Stephen Casperstephencasper.com
• History is Yours to Write - The Intelligence Curseintelligence-curse.ai
• Breaking the Intelligence Curse - The Intelligence Curseintelligence-curse.ai
• The Problem — LessWronglesswrong.com
• A Long Sequence of Small, Correct Decisionscalebwatney.substack.com
• The Post-AGI Workshop: Economics, Culture and Governance | San Diego 2025post-agi.org