[1706.04599] On Calibration of Modern Neural Networks

Abstract:Confidence calibration -- the problem of predicting probability estimates representative of the true correctness likelihood -- is important for classification models in many applications. We discover that modern neural networks, unlike those from a decade ago, are poorly calibrated. Through extensive experiments, we observe that depth, width, weight decay, and Batch Normalization are important factors influencing calibration. We evaluate the performance of various post-processing calibration methods on state-of-the-art architectures with image and document classification datasets. Our analysis and experiments not only offer insights into neural network learning, but also provide a simple and straightforward recipe for practical settings: on most datasets, temperature scaling -- a single-parameter variant of Platt Scaling -- is surprisingly effective at calibrating predictions.

# link_27qz6m7x2yr.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - Author=Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger - CreationDate=D:20170804011141Z - Creator=LaTeX with hyperref package - Keywords=calibration, confidence, deep learning, neural networks - ModDate=D:20170804011141Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.14159265-2.6-1.40.17 (TeX Live 2016) kpathsea version 6.2.2 - Producer=pdfTeX-1.40.17 - Subject=Proceedings of the International Conferenc

saved by

• Hangyul Lyna Kim
• Rikard Saqe
• Lydia Nottingham

related reading

• On Calibration of Modern Neural Networksproceedings.mlr.press
• On Getting Confidence Estimates from Neural Networks | Bharath's notesbharathpbhat.github.io
• The Decade of Deep Learning | Leo Gaobmk.sh
• On neural scaling and the quanta hypothesisericjmichaud.com
• Bayesian Neural Networkscs.toronto.edu
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• A Recipe for Training Neural Networkskarpathy.github.io
• [2003.07892] Calibration of Pre-trained Transformersarxiv.org
• arxiv.org/pdf/1805.08522arxiv.org
• The Little Book of Deep Learningfleuret.org
• Dealing with Overconfidence in Neural Networks: Bayesian Approach – Jonathan Ramkissoonjramkiss.github.io
• A Recipe for Training Neural Networkskarpathy.github.io