Imitative Generalisation (AKA 'Learning the Prior') — AI Alignment Forum
Tl;dr We want to be able to supervise models with superhuman knowledge of the world and how to manipulate it. For this we need an overseer to be able…
x Imitative Generalisation (AKA 'Learning the Prior') — AI Alignment Forum Debate (AI safety technique) Distillation & Pedagogy Iterated Amplification OpenAI Outer Alignment AI Frontpage 60 Imitative Generalisation (AKA 'Learning the Prior') by Beth Barnes 10th Jan 2021 13 min read 15 60 Tl;dr We want to be able to supervise models with superhuman knowledge of the world and how to manipulate it. For this we need an overseer to be able to learn or access all the knowledge our models have, in order to be able to understand the consequences of suggestions or decisions from the model. If the overs
related reading
- Just Ask for Generalization | Eric Jangevjang.com
- Simulators — LessWronglesswrong.com
- Better priors as a safety problem — LessWronglesswrong.com
- You can’t imitation-learn how to continual-learn — LessWronglesswrong.com
- Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
- To Understand Language is to Understand Generalization | Eric Jangevjang.com
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
- Behavior Cloning is Miscalibrated — AI Alignment Forumalignmentforum.org
- How far does alignment midtraining generalize?alignment.openai.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Externalized reasoning oversight: a research direction for language model alignment — AI Alignment Forumalignmentforum.org
- How To Go From Interpretability To Alignment: Just Retarget The Search — LessWronglesswrong.com