A Gentle Introduction to Cross-Entropy for Machine Learning - MachineLearningMastery.com
Cross-entropy is commonly used in machine learning as a loss function. Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions. It is closely related to but is different from KL divergence that calculates the relative entropy between two probability distributions, whereas cross-entropy […]
By Jason Brownlee on December 22, 2020 in Probability 74 Share Post Share Cross-entropy is commonly used in machine learning as a loss function. Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions. It is closely related to but is different from KL divergence that calculates the relative entropy between two probability distributions, whereas cross-entropy can be thought to calculate the total entropy between the distributions. Cross-entropy is also related to and often confused with
related reading
- Things that confused me about cross-entropychris-said.io
- Kullback–Leibler divergence - Wikipediaen.wikipedia.org
- Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
- Cross-entropy | Anıl Zenginoğluanilzen.github.io
- Approximating KL Divergencejoschu.net
- Visual Information Theory -- colah's blogcolah.github.io
- Maximum Entropy Distributions | Bounded Rationalitybjlkeng.github.io
- blog.alexalemi.com Why KL?blog.alexalemi.com
- What is the difference between a loss function and an error function? - Cross Validatedstats.stackexchange.com
- Label Smoothing Explained using Microsoft Excelamaarora.github.io
- Andy Jonesandrewcharlesjones.github.io
- Understanding softmax and the negative log-likelihoodljvmiranda921.github.io