Unfamiliar Finetuning Examples Control How Language Models Hallucinate

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions.

Unfamiliar Finetuning Examples Control How Language Models Hallucinate Katie Kang 1 , Eric Wallace 1 , Claire Tomlin 1 , Aviral Kumar 2 , Sergey Levine 1 ( 1 UC Berkeley 2 Google DeepMind) Abstract Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models’ finetuning data – those that introduce concepts beyond the base model’s scope of knowledge – are crucial in shaping these errors. In particular, we find that an LL

saved by

• Lawrence Feng

related reading

• Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelsarxiv.org
• LoRA vs Full Fine-tuning: An Illusion of Equivalencearxiv.org
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc
• Features as Rewards: Using Interpretability to Reduce Hallucinationsgoodfire.ai
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
• Alignment faking in large language modelsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org