AI existential risk probabilities are too unreliable to inform policy

How speculation gets laundered through pseudo-quantification

How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions. This is the first in a series of essays laying out an evidence-based approach for policymakers concerned about AI x-risk, an approach that stays grounded in reality while acknowledging that there are “unknown…

related reading

• Eight reasons that existential risk estimates are so uncertaintitotal.substack.com
• Notes on Existential Risk from Artificial Superintelligencemichaelnotebook.com
• Existential Risk from AI: An Exposition for Mathematiciansalkjash.github.io
• Where I agree and disagree with Eliezer — LessWronglesswrong.com
• The Artificiality of Alignmentjoinreboot.org
• Beyond P(doom) for AI Risk: Quantifying Uncertainty Without Probability | Center for Security and Emerging Technology Georgetown AIcset.georgetown.edu
• Predictable updating about AI riskjoecarlsmith.com
• My views on “doom” — LessWronglesswrong.com
• AI x-risk, approximately ordered by embarrassment — LessWronglesswrong.com
• [2206.13353] Is Power-Seeking AI an Existential Risk?arxiv.org
• Many arguments for AI x-risk are wrong — AI Alignment Forumalignmentforum.org
• Post 45: Simplify EA Pitches to “Holy Shit, X-Risk” - Neel Nandaneelnanda.io