What is neuralese and why is everyone so concerned about it?
transformernews.ai · 1,345 words · saved by 1 readers
OpenAI’s new Astra model is raising concerns about our continued ability to monitor AI’s chain of thought
Image: Jacob Wackerhausen / Getty Images Off the back of a report in The Information on Tuesday, everyone’s suddenly very worried about “neuralese”. OpenAI’s new Astra model, the outlet reported, was built with a new architecture which could make it harder to monitor its reasoning, raising safety concerns. Ryan Greenblatt, a prominent AI safety researcher, said that if true, the news “may be the single worst development for AI security/safety to date”. But what does any of this mean — and is the latest development really as bad as some have made it out to be? Let’s start with the very…
saved by
related reading
- Astra Is Hard to Monitorthezvi.substack.com
- AI #184: Post Post Mortemthezvi.substack.com
- How AI Is Learning to Think in Secretnickandresen.substack.com
- The Specter Of Neuraleseastralcodexten.com
- Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack? — LessWronglesswrong.com
- Astra for Coding: Why Are We Doing This Again?lucumr.pocoo.org
- GPT-6 Astra System Carddeploymentsafety.openai.com
- Cognitive Security as an AI Safety Cause Area — LessWronglesswrong.com
- The fragile foundations of CoT monitoring | Christopher Pottsweb.stanford.edu
- [2507.11473] Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safetyarxiv.org
- God Help Us, Let's Try To Understand The Paper On AI Monosemanticityastralcodexten.com
- An Alien Mind: Jakub Pachocki Warns Usthezvi.substack.com