A Gentle Introduction to Cross-Entropy for Machine Learning - MachineLearningMastery.com
Cross-entropy is commonly used in machine learning as a loss function. Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions. It is closely related to but is different from KL divergence that calculates the relative entropy between two probability distributions, whereas cross-entropy […]
By Jason Brownlee on December 22, 2020 in Probability 74 Share Post Share Cross-entropy is commonly used in machine learning as a loss function. Cross-entropy is a measure from the field of information theory, building upon entropy and generally calculating the difference between two probability distributions. It is closely related to but is different from KL divergence that calculates the relative entropy between two probability distributions, whereas cross-entropy can be thought to calculate the total entropy between the distributions. Cross-entropy is also related to and often confused with
Explore this link on the map →related reading
- Cross-entropy | Anıl Zenginoğluanilzen.github.io
- Kullback–Leibler divergence - Wikipediaen.wikipedia.org
- Visual Information Theory -- colah's blogcolah.github.io
- Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
- blog.alexalemi.com Why KL?blog.alexalemi.com
- Cut Cross Entropy from first principles - Omkaar Kamathomkaark.com
- Visual Information Theory -- colah's blogcolah.github.io
- Approximating KL Divergencejoschu.net
- Binary entropy function - Wikipediaen.wikipedia.org
- Maximum Entropy Distributions | Bounded Rationalitybjlkeng.github.io
- A Mathematical Theory of Communicationpeople.math.harvard.edu
- Short Notes on Divergence Measuresdanilorezende.com