AGI and Lock-in | Forethought

forethought.org · 16,675 words · saved by 1 readers

The long-term future of intelligent life is currently unpredictable and undetermined. We argue that the invention of artificial general intelligence (AGI) could change this by making extreme types of lock-in technologically feasible. In particular, we argue that AGI would make it technologically feasible to (i) perfectly preserve nuanced specifications of a wide variety of values or goals far into the future, and (ii) develop AGI-based institutions that would (with high probability) competently pursue any such values for at least millions, and plausibly trillions, of years.

AGI and Lock-in Lukas Finnveden Jess Riedel Carl Shulman Authors Citations Cite Citations PDF Contact 8th October 2022 Last update: 11th March 2025 AGI and Lock-in Abstract 0 Summary 0.0 The claim 0.1 Preserving information 0.2 Executing intentions 0.3 Preventing disruption 0.4 Some things we don’t argue for 0.5 Structure of the document 1 To be more precise 1.1 What do we assume about “AGI”? 1.2 What do we mean by “lock-in”? 1.3 How confident are we that stability is feasible, for how long? 2 Desirability and probability 2.1 Would it be good for highly stable institutions to be built? 2.2 How

AGI and Lock-in | Forethought

related reading