What training data should developers filter to reduce risk from misaligned AI?

One potentially powerful way to change the properties of AI models is to change their training data. For example, Anthropic has explored filtering training data to mitigate bio misuse risk. What data, if any, should be filtered to reduce misalignment risk? In this post, I argue that the highest ROI data to filter is information about safety measures, and strategies for subverting them. Here’s why this is valuable: If AI models don’t know what our security measures are, then they will need to either (1) guess them, (2) figure out what they are, or (3) iterate against them. But (1) seems hard, and (2) / (3) are good opportunities to catch the AI. If AI models aren’t trained on documents discussing how to subvert security measures, then they’d need to figure out for themselves how to subvert security measures. This is worse for the AI models, because they might make mistakes and we might catch them investigating these topics. In order to develop stronger safety measures, safety researcher

One potentially powerful way to change the properties of AI models is to change their training data. For example, Anthropic has explored filtering training data to mitigate bio misuse risk. What data, if any, should be filtered to reduce misalignment risk? In this post, I argue that the highest ROI data to filter is information about safety measures, and strategies for subverting them. Here’s why this is valuable: If AI models don’t know what our security measures are, then they will need to either (1) guess them, (2) figure out what they are, or (3) iterate against them. But (1) seems hard, a