Latent Adversarial Training — LessWrong

We’d like to train models to be robustly safe, even in environments that may fall well outside of the training distribution. Unfortunately all we get to work with is the training distribution, which makes ensuring robust generalization difficult. Deception is an example of this concern. Models may be well-behaved in the training environment because they realize it is a training environment, but behave poorly in deployment because they notice the distributional shift. It would be great if we could generate scenarios where the model behaves badly. For instance if the model is deceptive we’d like to generate inputs to trick it into thinking it’s been deployed so that it reveals that deception. Adversarial Training (AT), and Relaxed Adversarial Training (RAT), are approaches to this problem. Very briefly, in AT, we search for inputs that generate large loss, while in RAT we search for distributions of inputs or other more general evidence that the model is capable of generating large loss.

x Latent Adversarial Training — LessWrong Adversarial Training Deception AI Frontpage 58 Latent Adversarial Training by Adam Jermyn 29th Jun 2022 AI Alignment Forum 6 min read 13 58 Ω 25 The Problem We’d like to train models to be robustly safe, even in environments that may fall well outside of the training distribution. Unfortunately all we get to work with is the training distribution, which makes ensuring robust generalization difficult. Deception is an example of this concern. Models may be well-behaved in the training environment because they realize it is a training environment, but beh

related reading

• [2401.05566] Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingarxiv.org
• [2407.15549] Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMsarxiv.org
• Recommendations for Technical AI Safety Research Directionsalignment.anthropic.com
• [2604.22082] Removing Sandbagging in LLMs by Training with Weak Supervisionarxiv.org
• Relaxed adversarial training for inner alignment — AI Alignment Forumalignmentforum.org
• 39 - Evan Hubinger on Model Organisms of Misalignment | AXRP - the AI X-risk Research Podcastaxrp.net
• Should We Train Against (CoT) Monitors? — LessWronglesswrong.com
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• [2312.06942] AI Control: Improving Safety Despite Intentional Subversionarxiv.org
• Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• 2312.06942arxiv.org
• Advice for making robust-to-training model organismsblog.redwoodresearch.org