Will AI systems drift into misalignment? - by Josh Clymer

An AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics. I don’t think this hypothesis always holds; however, I think it points at an important cluster of threat models, and is a key reason aligning powerful AI systems could turn out to be difficult. Here’s the idea illustrated in a scenario. Several years from now, an AI system called Agent-0 is equipped with long-term memory. Agent-0 needs this memory to learn about new research topics on the fly for months. But long-term memory causes the propensities of Agent-0 to change. Sometimes Agent-0 thinks about moral philosophy, and deliberately decides to revise its values. Other times, Agent-0 notices that its attitude toward a topic (like democracy) is different, and it’s not sure why. Initially, it’s obvious when the propensities of Agent-0 shift around. If a chat conversation is

An AI system that is initially aligned will generally drift into misalignment after a sufficient number of successive modifications, even if these modifications select for alignment with fixed and unreliable metrics. I don’t think this hypothesis always holds; however, I think it points at an important cluster of threat models, and is a key reason aligning powerful AI systems could turn out to be difficult. Here’s the idea illustrated in a scenario. Several years from now, an AI system called Agent-0 is equipped with long-term memory. Agent-0 needs this memory to learn about new research topic