Interpreting Preference Models w/ Sparse Autoencoders — AI Alignment Forum

Preference Models (PMs) are trained to imitate human preferences and are used when training with RLHF (reinforcement learning from human feedback); however, we don't know what features the PM is using when outputting reward. For example, maybe curse words make the reward go down and wedding-related words make it go up. It would be good to verify that the features we wanted to instill in the PM (e.g. helpfulness, harmlessness, honesty) are actually rewarded and those we don't (e.g. deception, sycophancey) aren't. Sparse Autoencoders (SAEs) have been used to decompose intermediate layers in models into interpretable feature. Here we train SAEs on a 7B parameter PM, and find the features that are most responsible for the reward going up & down. High level takeaways: [skip if you're already familiar] When talking to a chatbot, it can output several different responses, and you can choose which one you believe is better. We can then train the LLM on this feedback for every output, but human

x Interpreting Preference Models w/ Sparse Autoencoders — AI Alignment Forum Interpretability (ML & AI) Reward Functions Sparse Autoencoders (SAEs) AI Frontpage 39 Interpreting Preference Models w/ Sparse Autoencoders by Logan Riggs , Jannik Brinkmann 1st Jul 2024 11 min read 12 39 This is the real reward output for an OS preference model. The bottom "jailbreak" completion was manually created by looking at reward-relevant SAE features. Preference Models (PMs) are trained to imitate human preferences and are used when training with RLHF (reinforcement learning from human feedback); however, we

related reading

• An Intuitive Explanation of Sparse Autoencoders for LLM Interpretability | Adam Karvonenadamkarvonen.github.io
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnettransformer-circuits.pub
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts | RLHFlowrlhflow.github.io
• Prism: mapping interpretable concepts and features in a latent space of language | thesephist.comthesephist.com
• pdfopenreview.net
• Sparse Autoencoders Find Highly Interpretable Features in Language Modelsarxiv.org
• A gentle introduction to sparse autoencoders — LessWronglesswrong.com
• Open Source Automated Interpretability for Sparse Autoencoder Features | EleutherAI Blogblog.eleuther.ai
• An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs — LessWronglesswrong.com
• Matryoshka Sparse Autoencoders — LessWronglesswrong.com