[2506.09501] Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference

Abstract:Large Language Models (LLMs) are now integral across various domains and have demonstrated impressive performance. Progress, however, rests on the premise that benchmark scores are both accurate and reproducible. We demonstrate that the reproducibility of LLM performance is fragile: changing system configuration, such as evaluation batch size, GPU count, and GPU version, can introduce significant differences in the generated responses. This issue is especially pronounced in reasoning models, where minor rounding differences in early tokens can cascade into divergent chains of thought, ultimately affecting accuracy. For instance, under bfloat16 precision with greedy decoding, a reasoning model like DeepSeek-R1-Distill-Qwen-7B can exhibit up to 9% variation in accuracy and 9,000 tokens difference in response length due to differences in GPU count, type, and evaluation batch size. We trace the root cause of this variability to the non-associative nature of floating-point arithmetic under limited numerical precision. This work presents the first systematic investigation into how numerical precision affects reproducibility in LLM inference. Through carefully controlled experiments across various hardware, software, and precision settings, we quantify when and how model outputs diverge. Our analysis reveals that floating-point precision - while critical for reproducibility - is often neglected in evaluation practices. Inspired by this, we develop a lightweight inference pipeline, dubbed LayerCast, that stores weights in 16-bit precision but performs all computations in FP32, balancing memory efficiency with numerical stability. Code is available at this https URL.

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference Jiayi Yuan1∗ Hao Li2∗ Xinheng Ding2 Wenya Xie2 Yu-Jhe Li3 Wentian Zhao3 Kun Wan3 Jing Shi3 Xia Hu1 Zirui Liu2 1 Rice University 2 University of Minnesota Twin Cities 3 Adobe Inc.…

related reading

• Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
• [2506.09501] Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencearxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Non-Determinism of “Deterministic” LLM Settingsarxiv.org
• Defeating the Training-Inference Mismatch via FP16arxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• [2412.00543] Evaluating the Consistency of LLM Evaluatorsarxiv.org
• Large Language Diffusion Modelsarxiv.org
• [2506.17209] Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencyarxiv.org
• [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
• [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
• [2402.12374] Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingarxiv.org