A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
arxiv.org · 6,398 words · saved by 1 readers
N/A
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity Andrew Lee 1 Xiaoyan Bai 1 Itamar Pres 1 Martin Wattenberg 2 Jonathan K. Kummerfeld 3 Rada Mihalcea 1 Abstract our understanding of the mechanisms by which the unde- sirable behavior is…
saved by
related reading
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Mechanistically Eliciting Latent Behaviors in Language Models — AI Alignment Forumalignmentforum.org
- Position: It's Time to Optimize for Self-Consistencytime-for-consistency.github.io
- 2308.03958arxiv.org
- [2602.04899] Phantom Transfer: Data-level Defences are Insufficient Against Data Poisoningarxiv.org
- Alignment faking in large language modelsarxiv.org
- [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
- Auditing language models for hidden objectives — LessWronglesswrong.com
- [2601.10160] Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentarxiv.org