Understanding and controlling a maze-solving policy network — AI Alignment Forum

TL;DR: We algebraically modified the net's runtime goals without finetuning. We also found (what we think is) a "motivational API" deep in the network. We used the API to retarget the agent. Summary of a few of the most interesting results: Langosco et al. trained a range of maze-solving nets. We decided to analyze one which we thought would be interesting. The network we chose has 3.5M parameters and 15 convolutional layers. In this post, we'll mostly discuss what we found, not what our findings mean. Let's run through some facts about Langosco et al.'s training process. Mazes had varying effective sizes, ranging from 3 × 3 to 25 × 25 : Each 64 × 64 RGB observation is processed by a deeply convolutional (15 conv layers!) network, without memory (i.e. no recurrent state): Why does the agent go to the cheese sometimes, and the top-right corner other times? It's not that the agent wasn't trained for long enough. Sampling rollouts from the trained policy adds a lot of noise. It's als

x Understanding and controlling a maze-solving policy network — AI Alignment Forum Interpreting a Maze-Solving Network MATS Program Shard Theory Activation Engineering Inner Alignment Interpretability (ML & AI) AI Frontpage 140 Understanding and controlling a maze-solving policy network by TurnTrout , peligrietzer , Ulisse Mini , Monte M , David Udell 11th Mar 2023 27 min read 28 140 Previously: Predictions for shard theory mechanistic interpretability results Locally retargeting the search by modifying a single activation. We found a residual channel halfway through a maze-solving network. Wh

related reading

• Zoom In: An Introduction to Circuitsdistill.pub
• Learning Beyond Gradientstrinkle23897.github.io
• Reward is not the optimization target — LessWronglesswrong.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• pdfopenreview.net
• Models Don't "Get Reward" — LessWronglesswrong.com
• Jane Street Blog - Can you reverse engineer our neural network?blog.janestreet.com
• AlgZoo: uninterpreted models with fewer than 1,500 parameters — LessWronglesswrong.com
• MordatchNIPS15.pdfroboti.us
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationarxiv.org
• Deep Reinforcement Learning: Pong from Pixelskarpathy.github.io
• Understanding RL Visiondistill.pub