[2606.18327] Self-CTRL: Self-Consistency Training with Reinforcement Learning

Abstract:Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of biased samplers and evaluated on their ability to report the associated biases. We find that consistency training improves the correlation between self-reported and behaviorally-measured latent biases from $R^2=0.24$ to $R^2=0.64$ on a set of held-out distributions, matching the generalization of direct ground-truth supervision. Second, we study a constitutional AI domain in which LMs must describe when they will refuse or comply with user requests. Here, Self-CTRL produces rules that faithfully describe the model's behavior on held-out requests, improving the refusal predictions of a third-party auditor model from $36\%$ to $92\%$. In the other direction, behavior updates improve alignment, reducing HarmBench failure rate from $15.0\%$ to $0.5\%$ without substantially increasing refusal on harmless prompts. By aligning explanations and behavior, our work provides a general recipe for training AI models to be safer, more transparent, and more controllable.

View PDF HTML (experimental) Abstract:Language models (LMs) that faithfully describe their own behavior can more easily be audited, understood, and trusted by users. This paper describes Self-Consistency Training with Reinforcement Learning (Self-CTRL), a method that optimizes for consistency between a LM's self-explanations and behavior on related inputs by updating explanations to better predict behavior or updating behavior to better match explanations. We apply our method in two domains. First, we study a formal probabilistic reasoning task in which LMs must learn to imitate a family of…

saved by

• Lydia Nottingham
• Anonymous Cow

related reading

• [2501.11120] Tell me about yourself: LLMs are aware of their learned behaviorsarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Position: It's Time to Optimize for Self-Consistencytime-for-consistency.github.io
• Foundation Models for Oversight | Transluce AItransluce.org
• Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• Unsupervised Elicitation of Language Modelsarxiv.org
• [2604.16812] Introspection Adapters: Training LLMs to Report Their Learned Behaviorsarxiv.org
• Why We Are Excited About Confessionsalignment.openai.com
• How confessions can keep language models honest | OpenAIopenai.com
• [2510.27062] Consistency Training Helps Stop Sycophancy and Jailbreaksarxiv.org