Oversight Assistants: Turning Compute into Understanding

Currently, we primarily oversee AI with human supervision and human-run experiments, possibly augmented by off-the-shelf AI assistants like ChatGPT or Claude. At training time, we run RLHF, where humans (and/or chat assistants) label behaviors with whether they are good or not. Afterwards, human researchers do additional testing to surface

Currently, we primarily oversee AI with human supervision and human-run experiments, possibly augmented by off-the-shelf AI assistants like ChatGPT or Claude. At training time, we run RLHF , where humans (and/or chat assistants ) label behaviors with whether they are good or not. Afterwards, human researchers do additional testing to surface and evaluate unwanted behaviors, possibly assisted by a scaffolded chat agent . The problem with primarily human-driven oversight is that it is not scalable: as AI systems keep getting smarter, errors become harder to detect: The behaviors we care about be

related reading

• Oversight Assistants: Turning Compute into Understandingbounded-regret.ghost.io
• Measuring progress on scalable oversightanthropic.com
• Foundation Models for Oversight | Transluce AItransluce.org
• Alignment is not solved but it increasingly looks solvablealigned.substack.com
• Automated Alignment Researchers: Using large language models to scale scalable oversight \ Anthropicanthropic.com
• Scaling Activation Oracles to Trillion-Parameter Modelstransluce.org
• Constitutional AI: Harmlessness from AI Feedbackarxiv.org
• The case for ensuring that powerful AIs are controlled — LessWronglesswrong.com
• Can we scale human feedback for complex AI tasks? An intro to scalable oversight.aisafetyfundamentals.com
• Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover — LessWronglesswrong.com
• AI Control: Improving Safety Despite Intentional Subversion — LessWronglesswrong.com
• Auditing language models for hidden objectives — LessWronglesswrong.com