The world's first frontier AI regulation is surprisingly thoughtful: the EU's Code of Practice

We’ve previously written about what an individual can do to make the development of transformative AI less likely to end in disaster. How about an AGI company?1 What steps should they take right now to prepare for crunch time? The first thing we’d recommend an AGI company do is to coordinate with other companies and with governments to stop the reckless race toward superintelligence. Failing that, our backup recommendation would be for an AGI company to invest in planning and transparency. We expect that during takeoff, leading AGI companies will have to make high-stakes decisions based on limited evidence under crazy time pressure. As depicted in AI 2027, the leading American AI company might have just weeks to decide whether to hand their GPUs to a possibly misaligned superhuman AI R&D agent they don’t understand. Getting this decision wrong in either direction could lead to disaster. Deploy a misaligned agent, and it might sabotage the development of its vastly superhuman successor.

We’ve previously written about what an individual can do to make the development of transformative AI less likely to end in disaster. How about an AGI company?1 What steps should they take right now to prepare for crunch time? The first thing we’d recommend an AGI company do is to coordinate with other companies and with governments to stop the reckless race toward superintelligence. Failing that, our backup recommendation would be for an AGI company to invest in planning and transparency. We expect that during takeoff, leading AGI companies will have to make high-stakes decisions based on lim