Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Training - Goodfire
Post-training can introduce undesired side effects that are difficult to detect and even harder to trace to specific training datapoints. We show that a probe-based method can surface concerning behaviors that emerge during LLM post-training, and that probes can identify the datapoints responsible for a specific harmful behavior. Filtering out those datapoints and retraining significantly reduces the behavior.
Introduction Post-training can introduce undesired side effects that are difficult to detect and even harder to trace to specific training datapoints. We show that a probe-based method can surface concerning behaviors that emerge during LLM post-training, and that probes can identify the datapoints responsible for a specific harmful behavior. Filtering out those datapoints and retraining significantly reduces the behavior. We introduce a natural testbed for data attribution: a harmful behavior that emerges during DPO training of OLMo 2 7B, where the model learns to comply with certain…
saved by
related reading
- Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivesarxiv.org
- Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
- How to build fast, efficient monitors for AI models using probes - Goodfiregoodfire.com
- Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
- [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org