Andy Jones

andrewcharlesjones.github.io · 1,769 words · saved by 1 readers

Recall that the reverse KL divergence between two distributions, q ( x ) 𝑞 ( 𝑥 ) and p ( x ) 𝑝 ( 𝑥 ) is defined as Expanding the log, we can write this in terms of an entropy term and a cross entropy term E q ( x ) [ log q ( x ) ] − E q ( x ) [ log p ( x ) ] = − H ( q ) + H c ( q , p ) 𝐸 𝑞 ( 𝑥 ) [ log ⁡ 𝑞 ( 𝑥 ) ] − 𝐸 𝑞 ( 𝑥 ) [ log ⁡ 𝑝 ( 𝑥 ) ] = − 𝐻 ( 𝑞 ) + 𝐻 𝑐 ( 𝑞 , 𝑝 ) If we can estimate the entropy of q 𝑞 exactly, then all that’s left is to get a Monte Carlo estimate of E q ( x ) [ log p ( x ) ] 𝐸 𝑞 ( 𝑥 ) [ log ⁡ 𝑝 ( 𝑥 ) ] . We can do this by sampling x 1 , … , x T ∼ q ( x ) 𝑥 1 , … , 𝑥 𝑇 ∼ 𝑞 ( 𝑥 ) and computing There are two types of KL divergence: forward and reverse. Forward KL divergence corresponds to D K L ( p | q ) 𝐷 𝐾 𝐿 ( 𝑝 | 𝑞 ) , and reverse corresponds to D K L ( q | p ) 𝐷 𝐾 𝐿 ( 𝑞 | 𝑝 ) . The reverse KL divergence is said to be “mode-seeking”. This means that the divergence will be low when q 𝑞 places densi

Andy Jones //--> $KL(q | p)$ is mode-seeking | Andy Jones $KL(q \| p)$ is mode-seeking Here, we discuss and visualize the mode-seeking behavior of the reverse KL divergence. Introduction Recall that the reverse KL divergence between two distributions, $q(x)$ and $p(x)$ is defined as \[D_{KL}(q(x) \| p(x)) = \mathbb{E}_{q(x)}\left[ \log \frac{q(x)}{p(x)} \right] = \int_{\mathbb{X}} \log \frac{q(x)}{p(x)} q(x) dx.\] Expanding the log, we can write this in terms of an entropy term and a cross entropy term \begin{align} &\mathbb{E}_{q(x)}\left[ \log q(x) \right] - \mathbb{E}_{q(x)}\left[ \log p(x)

saved by

Alexiy Buynitsky

Andy Jones

saved by

related reading