Cycles of Thought: Measuring LLM Confidence through Stable Explanations

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. \ul In many critical machine learning (ML) applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods for ML uncertainty quantification can be difficult to directly adapt to LLMs due to the computational cost of implementation and closed-source nature of many models. A variety of black-box methods have recently been proposed, but these often rely on heuristics such as self-verbalized confidence. We instead propose a framework for measuring an LLM’s uncertainty with respect to the distribution of generated ex

\useunder \ul Cycles of Thought: Measuring LLM Confidence through Stable Explanations Evan Becker Department of Computer Science UCLA evbecker@ucla.edu &Stefano Soatto Department of Computer Science UCLA soatto@cs.ucla.edu (October 16, 2025) Abstract In many critical machine learning (ML) applications it is essential for a model to indicate when it is uncertain about a prediction. While large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, their overconfidence in incorrect responses is still a well-documented failure mode. Traditional methods

related reading

• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2201.11903] Chain of Thought Prompting Elicits Reasoning in Large Language Modelsarxiv.org
• [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
• [2605.27288] It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintyarxiv.org
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org
• Non-Determinism of “Deterministic” LLM Settingsarxiv.org
• Making LLMs Say What They Think: Measuring and Improving CoT-Interpretability Alignmentarxiv.org
• Training Language Models to Explain Their Own Computationsarxiv.org
• Position: It's Time to Optimize for Self-Consistencytime-for-consistency.github.io
• Training Language Models to Explain Their Own Computationsarxiv.org
• Evaluation-Conditioned Training: Teaching Models to Generalize to Stronger Oversight Regimesarxiv.org
• Prediction, Explanation, or Over-interpretation?elena-baixy.github.io