flâneur

Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWrong

lesswrong.com · 4,520 words · saved by 2 readers

This doubled as my MATS application, felt like posting it here because the results are quite interesting. (github repo with code) Activation oracles (iterating on LatentQA) are an interpretability technique, capable of generating natural language explanations about activations with surprising generality. How robust are these oracles? Can we find a vector that maximises confidence of the oracle (in token probability) that a concept is represented, and can we then use this vector to steer the model? Is it possible to find feature representations that convince the oracle a concept is being represented, when it really is random noise? Effectively finding a counterexample to the oracle? (This would be bad in a world where we rely on them for truth). I provide at least one example where this is the case, finding 2 vectors that satisfy the oracle with one influencing causal behavior and the other doing nothing. (I can find vectors that fool the oracle, yet have a MSE of < 1.on final layer on

x Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs — LessWrong AI Frontpage 23 Dreaming Vectors: Gradient-descented steering vectors from Activation Oracles and using them to Red-Team AOs by ceselder 23rd Dec 2025 14 min read 4 23 This doubles as my Neel stream MATS application, figured I would crosspost it to LW because the results are interesting EDIT: Got accepted! :) Executive summary What problem am I trying to solve?/TLDR Activation oracles (iterating on LatentQA ) are an interpretability technique, capable of generating natural

saved by

related reading