Lessons from building an automated research scaffold — LessWrong

TL;DR. We built a scaffold to speed up our own research and gather data on automated alignment research (AAR). It turned out to not be valuable for r…

TL;DR. We built a scaffold to speed up our own research and gather data on automated alignment research (AAR). It turned out to not be valuable for researcher uplift, but was useful for gathering certain failure modes of AAR. Going forward, we plan to study the broader failure modes of AAR and how these automated research systems can be monitored and analyzed. We’d like to thank Sid Baines, Andrew Draganov, Cameron Holmes and Daniel Tan for helpful comments. This work was carried out by the Alignment Team at Arcadia Impact in collaboration with Josh Hills, Falko Galperin, and Denis Lim…

saved by

• Abhay Sheshadri
• Ishaan Panigrahi

related reading

• Automated Weak-to-Strong Researcheralignment.anthropic.com
• Automated alignment is harder than you thinkarxiv.org
• Alignment is not solved but it increasingly looks solvablealigned.substack.com
• “Alignment Engineering” vs. “Misalignment Science”lesswrong.com
• Tips for Empirical Alignment Research — AI Alignment Forumalignmentforum.org
• Automated Alignment Researchers: Using large language models to scale scalable oversight \ Anthropicanthropic.com
• Automated Alignment is Harder Than You Think — LessWronglesswrong.com
• Research Lessons I Learned at MATS & Redwoodlesswrong.com
• Can we safely automate alignment research? - Joe Carlsmithjoecarlsmith.com
• A basic systems architecture for AI agents that do autonomous research — LessWronglesswrong.com
• Towards self-driving codebases · Cursorcursor.com
• Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org