Jo Jiao, Boyd Kane - Limited-parameter finetuning [MATS Symposium | Winter 2026] - YouTube
MATS fellows Jo Jiao and Boyd Kane give a spotlight talk at the Winter 2026 MATS Symposium. Project Title: Limited-parameter finetuning can uncover misaligne...
MATS fellows Jo Jiao and Boyd Kane give a spotlight talk at the Winter 2026 MATS Symposium. Project Title: Limited-parameter finetuning can uncover misaligned LLMs Mentors: Alex Turner, Alex Cloud Brief Abstract: We detect whether a LLM is misaligned in various ways (sycophancy, reward hacking, etc.) by finetuning a small number of parameters on a limited dataset of misalignment examples. Apply to MATS and learn more: https://www.matsprogram.org It's 2027. Agent 2 is here. The model looks pretty aligned. It scores pretty green on behavioral evals. It tells us it loves animals and…
saved by
related reading
- Joan Velja, Lennie Wells - Building an empirical Science of AI Debate [MATS Symposium | Winter 2026]youtube.com
- Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
- Alignment is not solved but it increasingly looks solvablealigned.substack.com
- Alignment remains a hard, unsolved problem — LessWronglesswrong.com
- [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
- How far does alignment midtraining generalize?alignment.openai.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com
- Narrow Misalignment is Hard, Emergent Misalignment is Easy — LessWronglesswrong.com
- Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research — AI Alignment Forumalignmentforum.org
- How hard is it to inoculate against misalignment generalization? — LessWronglesswrong.com
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
- Model Spec Midtraining: Improving How Alignment Training Generalizesalignment.anthropic.com