Maximum Entropy Distributions | Bounded Rationality
bjlkeng.github.io · 2,663 words · saved by 1 readers
A introduction to maximum entropy distributions.
This post will talk about a method to find the probability distribution that best fits your given state of knowledge. Using the principle of maximum entropy and some testable information (e.g. the mean), you can find the distribution that makes the fewest assumptions about your data (the one with maximal information entropy). As you may have guessed, this is used often in Bayesian inference to determine prior distributions and also (at least implicitly) in natural language processing applications with maximum entropy (MaxEnt) classifiers (i.e. a multinomial logistic regression). As usual, I'll
related reading
- Principle of maximum entropy - Wikipediaen.wikipedia.org
- Maximum Entropy Methods (MaxEnt)bactra.org
- Visual Information Theory -- colah's blogcolah.github.io
- Six (and a half) intuitions for KL divergence — LessWronglesswrong.com
- A Maximum Entropy Intuition for Fundamental Statistical Distributions | long intuitionlongintuition.com
- Information entropy (Foundations of information theory: Part 2) - Matthew N. Bernsteinmbernste.github.io
- [2206.07867] A visual introduction to information theoryarxiv.org
- Kullback–Leibler divergence - Wikipediaen.wikipedia.org
- Introduction to abstract entropy — LessWronglesswrong.com
- Cross-entropy | Anıl Zenginoğluanilzen.github.io
- Reinventing Entropy | Compression is Intelligence Part 1youtube.com
- Things that confused me about cross-entropychris-said.io