Synthetic Persona Pretraining: Alignment from Token Zero
Installing the desired assistant persona from the first token of pretraining improves constitution following, value alignment, and jailbreak robustness, and the advantage grows with pretraining budget.
TL;DR. Alignment, and the assistant identity itself, are typically introduced only after pretraining, once behavioral priors are already established, so that values are a thin overlay, rather than deeply rooted. We propose Synthetic Persona Pretraining (SPP), installing the desired assistant persona from token zero in pretraining: we annotate pretraining documents with first-person moral reflections derived from a normative value constitution, pretrain on them, and then post-train to bind the assistant identity to the pretrained persona, a phenomenon we call persona binding. Pretraining…
saved by
related reading
- Persona Features Control Emergent Misalignmentarxiv.org
- Why study alignment interventions on pre-RL checkpoints? — LessWronglesswrong.com
- Spilling the Beans: Teaching LLMs to Self-Report Their Hidden Objectivesarxiv.org
- Synthetic Persona Pretraining: Alignment from Token Zero — LessWronglesswrong.com
- [2601.10160] Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentarxiv.org
- The Persona Selection Model: Why AI Assistants might Behave like Humansalignment.anthropic.com
- Dense, on-policy, or both?baseten.co
- Thoughts on the persona selection modellesswrong.com
- Reinforcement learning towards broadly and persistently beneficial modelsalignment.openai.com
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignmentalignmentpretraining.ai
- Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment — LessWronglesswrong.com
- Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Modelsturntrout.com