✳flâneur — a map of the web's best reading

How much novel security-critical infrastructure do you need during the singularity? — AI Alignment Forum

alignmentforum.org · 2,801 words · saved by 1 readers

I think a lot about the possibility of huge numbers of AI agents doing AI R&D inside an AI company (as depicted in AI 2027). I think particularly about what will happen if those AIs are scheming: coherently and carefully trying to grab power and take over the AI company, as a prelude to taking over the world. And even more particularly, I think about how we might try to mitigate the insider risk posed by these AIs, taking inspiration from traditional computer security, traditional insider threat prevention techniques, and first-principles thinking about the security opportunities posed by the differences between AIs and humans. So to flesh out this situation, I’m imagining a situation something like AI 2027 forecasts for March 2027: I want to know what we need to do to ensure that even if those AIs are conspiring against us, they aren’t able to cause catastrophic security failures.[1] One important question here is how strongly incentivized we are to use the AIs to write security-criti

x How much novel security-critical infrastructure do you need during the singularity? — AI Alignment Forum AI Takeoff Computer Security & Cryptography AI Frontpage 31 How much novel security-critical infrastructure do you need during the singularity? by Buck 4th Jul 2025 6 min read 7 31 I think a lot about the possibility of huge numbers of AI agents doing AI R&D inside an AI company (as depicted in AI 2027). I think particularly about what will happen if those AIs are scheming: coherently and carefully trying to grab power and take over the AI company, as a prelude to taking over the world. A

Explore this link on the map →

related reading