The case for countermeasures to memetic spread of misaligned values

Defending against alignment problems that might come with long-term memory

As various people have written about before, AIs that have long-term memory might pose additional risks (most notably, LLM AGI will have memory, and memory changes alignment by Seth Herd). Even if an AI is aligned or only occasionally scheming at the start of a deployment, the AI might become a consistent and coherent behavioral schemer via updates to its long-term memories. In this post, I’ll spell out the version of the threat model that I’m most concerned about, including some novel arguments for its plausibility, and describe some promising strategies for mitigating this risk. While I…

saved by

• Jo J.
• Dillon Nguyen

related reading

• The case for countermeasures to memetic spread of misaligned values — LessWronglesswrong.com
• The case for ensuring that powerful AIs are controlled — LessWronglesswrong.com
• The case for ensuring that powerful AIs are controlledblog.redwoodresearch.org
• The Case Against AI Control Research — LessWronglesswrong.com
• Current AIs seem pretty misaligned to meblog.redwoodresearch.org
• Thoughts on the conservative assumptions in AI controlblog.redwoodresearch.org
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
• Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
• Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
• Current AIs seem pretty misaligned to me — LessWronglesswrong.com
• Alignment Faking Mitigationsalignment.anthropic.com
• Agentic misalignment: How LLMs could be insider threats \ Anthropicanthropic.com