How well do truth probes generalise? — LessWrong

Representation engineering (RepEng) has emerged as a promising research avenue for model interpretability and control. Recent papers have proposed me…

x How well do truth probes generalise? — LessWrong Activation Engineering AI Frontpage 96 How well do truth probes generalise? by mishajw 24th Feb 2024 11 min read 11 96 Representation engineering (RepEng) has emerged as a promising research avenue for model interpretability and control. Recent papers have proposed methods for discovering truth in models with unlabeled data , guiding generation by modifying representations , and building LLM lie detectors . RepEng asks the question: If we treat representations as the central unit, how much power do we have over a model’s behaviour? Most techni

saved by

• Micah Carroll
• Asher P
• Davis Brown

related reading

• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io
• Taking LLMs Seriously (As Language Models) — LessWronglesswrong.com
• Probe-Based Data Attribution: Surfacing and Mitigating Undesirable Behaviors in LLM Post-Traininggoodfire.ai
• 2310.01405arxiv.org
• Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
• Can activation verbalizers surface an internal chain of thought? — LessWronglesswrong.com
• Do Machine Learning Models Memorize or Generalize?pair.withgoogle.com
• Features as Rewards: Using Interpretability to Reduce Hallucinationsgoodfire.ai
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Is This Lie Detector Really Just a Lie Detector? An Investigation of LLM Probe Specificity. — LessWronglesswrong.com
• Quantifying Truesight With SAEs · Gwern.netgwern.net
• [2310.06824] The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetsarxiv.org