SOTA alignment assessments don’t strongly update us against misalignment

Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1][2], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”

Anthropic concluded in the April Mythos Preview alignment risk update that the model “does not possess any unknown propensities that would increase alignment risk.” The report argues that if Mythos Preview were coherently misaligned[1][2], it likely would have been detected by the assessment (following Anthropic, I will call this “reliability of the assessment”[3]). While I agree with the report on the above bottom-line conclusions (substantially on priors)[4], I think there are gaps in its argument which weaken the current assessment and might invalidate future assessments. In particular,…

saved by

• Yixiong Hao
• Will Anderson
• Carolyn Qian

related reading

• Alignment is not solved but it increasingly looks solvablealigned.substack.com
• Alignment remains a hard, unsolved problem — LessWronglesswrong.com
• Current AIs seem pretty misaligned to me — LessWronglesswrong.com
• Automated Alignment Researchers: Using large language models to scale scalable oversight \ Anthropicanthropic.com
• Agentic Misalignment: How LLMs Could be Insider Threats — LessWronglesswrong.com
• The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn'tblog.redwoodresearch.org
• Are we existentially threatened by the type of AI misalignment seen in the OpenAI Hugging Face attack? — LessWronglesswrong.com
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
• Metagaming matters for training, evaluation, and oversightalignment.openai.com
• Pre-deployment auditing can catch an overt saboteuralignment.anthropic.com