flâneur

Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data

alignment.anthropic.com · 1,340 words · saved by 6 readers

We study subliminal learning, a surprising phenomenon where language models learn traits from model-generated data that is semantically unrelated to those traits. For example, a "student" model learns to prefer owls when trained on sequences of numbers generated by a "teacher" model that prefers owls. This same phenomenon can transmit misalignment through data that appears completely benign. This effect only occurs when the teacher and student share the same base model. 📄Paper, 💻Code Research done as part of the Anthropic Fellows Program. Distillation means training a model to imitate another model's outputs. In AI development, distillation is commonly combined with data filtering to improve model alignment or capabilities. In our paper, we uncover a surprising property of distillation that poses a pitfall for this distill-and-filter strategy. Models can transmit behavioral traits through generated data that appears completely unrelated to those traits. The signals that transmit thes

Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data Alignment Science Blog Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data Alex Cloud* 1 , Minh Le* 1 , July 22, 2025 James Chua 2 , Jan Betley 2 , Anna Sztyber-Betley 3 , Jacob Hilton 4 , Samuel Marks 5 , Owain Evans 2,6 *Equal contribution; author order chosen randomly 1 Anthropic Fellows Program; 2 Truthful AI; 3 Warsaw University of Technology; 4 Alignment Research Center; 5 Anthropic; 6 UC Berkeley tl;dr We study subliminal learning , a surprising phenomenon wh

saved by

related reading