Non-Determinism of “Deterministic” LLM Settings

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected to be deterministic. Yet the questions of how pervasive this is, and with what impact on results, have not to our knowledge been systematically investigated. We investigate non-determinism in five LLMs configured to be deterministic when applied to eight common tasks in across 10 runs, in both zero-shot and few-shot settings. We see accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%. In fact, none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings. Sharing preliminary results with insiders

Non-Determinism of “Deterministic” LLM Settings Berk Atil 1 , Sarp Aykent 2 , Alexa Chittams 2 , Lisheng Fu 2 , Rebecca J. Passonneau 1 , Evan Radcliffe 2 , Guru Rajan Rajagopal 2 , Adam Sloan 2 , Tomasz Tudrej 2 , Ferhan Ture 2 , Zhe Wu 2 , Lixinyu Xu 2 , Breck Baldwin 2 1 Penn State University, 2 Comcast AI Technologies Correspondence: {bka5352,rjp49}@psu.edu ; breckbaldwin@gmail.com Berk Atil completed this work during his internship at Comcast AI Technologies Abstract LLM (large language model) practitioners commonly notice that outputs can vary for the same inputs under settings expected

saved by

• Annie Zhou

related reading

• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• [2506.09501] Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencearxiv.org
• Large Language Diffusion Modelsarxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2605.27288] It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertaintyarxiv.org
• [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
• Are Emergent Abilities in Large Language Models just In-Context Learning?arxiv.org
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• Cycles of Thought: Measuring LLM Confidence through Stable Explanationsarxiv.org
• Predicting LLM Safety Before Release by Simulating Deploymentcdn.openai.com