Detecting when LLMs are Uncertain • Thariq Shihipar

A deep dive into a new reasoning technique called Entropix.

Detecting when LLMs are Uncertain Thariq Shihipar - 11 October 2024 · 7 min read This post tries to explain the new reasoning techniques developed by XJDR in a new project called Entropix . Entropix attempts to improve reasoning in models through being smarter at sampling during moments of uncertainty. A big caveat, there have been no large scale evals yet for Entropix, so it’s not clear how much this helps in practice. But it does seem to introduce some promising techniques and mental models for reasoning. Uncertainity at a glance Sampling is the process of choosing which token from the distr

related reading

• Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generationarxiv.org
• As Rocks May Think | Eric Jangevjang.com
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• Cycles of Thought: Measuring LLM Confidence through Stable Explanationsarxiv.org
• Towards a Typology of Strange LLM Chains-of-Thought1a3orn.com
• Tenobrus (@tenobrus) on Xx.com
• The Unintelligibility is Ours: Notes on Chain of Thought1a3orn.com
• [2605.27288] It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintyarxiv.org
• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• Confidence Regulation Neurons in Language Modelsarxiv.org
• Thought Branches: Interpreting LLM Reasoning Requires Resamplingarxiv.org
• Is AI Reasoning Right for the Wrong Reasons? | Quanta Magazinequantamagazine.org