AI existential risk probabilities are too unreliable to inform policy

How speculation gets laundered through pseudo-quantification

How seriously should governments take the threat of existential risk from AI, given the lack of consensus among researchers? On the one hand, existential risks (x-risks) are necessarily somewhat speculative: by the time there is concrete evidence, it may be too late. On the other hand, governments must prioritize — after all, they don’t worry too much about x-risk from alien invasions. This is the first in a series of essays laying out an evidence-based approach for policymakers concerned about AI x-risk, an approach that stays grounded in reality while acknowledging that there are “unknown…

related reading

• Beyond P(doom) for AI Risk: Quantifying Uncertainty Without Probability | Center for Security and Emerging Technology Georgetown AIcset.georgetown.edu
• Eight reasons that existential risk estimates are so uncertaintitotal.substack.com
• You will be OK — LessWronglesswrong.com
• Where I agree and disagree with Eliezer — LessWronglesswrong.com
• Existential Risk from AI: An Exposition for Mathematiciansalkjash.github.io
• Why I am not a longtermist – Windows On Theorywindowsontheory.org
• near-term-xpt-accuracy.pdfstatic1.squarespace.com
• Post 45: Simplify EA Pitches to “Holy Shit, X-Risk” - Neel Nandaneelnanda.io
• Predictable updating about AI riskjoecarlsmith.com
• Notes on Existential Risk from Artificial Superintelligencemichaelnotebook.com
• My views on “doom” — LessWronglesswrong.com
• The Artificiality of Alignmentjoinreboot.org