flâneur

Tuan Anh Le

tuananhle.co.uk · 458 words · saved by 1 readers

Let p(x)= ∑ k=1 K π k Normal(x; μ k , σ 2 k ) q ϕ (x)=Normal(x; μ q , σ 2 q ), (1) (2) (1) 𝑝 ( 𝑥 ) = ∑ 𝑘 = 1 𝐾 𝜋 𝑘 N o r m a l ( 𝑥 ; 𝜇 𝑘 , 𝜎 𝑘 2 ) (2) 𝑞 𝜙 ( 𝑥 ) = N o r m a l ( 𝑥 ; 𝜇 𝑞 , 𝜎 𝑞 2 ) , where K=2 𝐾 = 2 , π k =0.5, σ 2 k =1 𝜋 𝑘 = 0.5 , 𝜎 𝑘 2 = 1 ( k=1,2 𝑘 = 1 , 2 ), μ 1 =0 𝜇 1 = 0 , μ 2 𝜇 2 is increasing from 0 0 to 10 10 and ϕ=( μ q , σ 2 q ) 𝜙 = ( 𝜇 𝑞 , 𝜎 𝑞 2 ) . The behavior of minimizing the forward and reverse KL divergences with respect to q ϕ 𝑞 𝜙 is as follows: Intuitive explanations and speculations for the origins of the various terms: The terms mode-seeking and zero-forcing for minimizing the reverse KL comes from the fact that this minimization forces q ϕ 𝑞 𝜙 to be zero where p 𝑝 is zero and hence makes it concentrate on one of the modes (last two plots). While the zero-forcing behavior can be explained by looking at the expression for the reverse KL divergence (when p 𝑝 is (almost) zero and q ϕ 𝑞 𝜙 is no

Reverse vs Forward KL 17 December 2017 Consider \(q_\phi(x)\), a probability density of \(x\), parametrized by \(\phi\), and \(p(x)\), a probability density of \(x\). Then If we minimize the \(\KL{q_\phi}{p}\) (or the reverse/exclusive KL) with respect to \(\phi\), the zero-forcing/mode-seeking behavior arises. If we minimize the \(\KL{p}{q_\phi}\) (or the forward/inclusive KL) with respect to \(\phi\), the mass-covering/mean-seeking behavior arises. Gaussian Example Let \begin{align} p(x) = \sum_{k = 1}^K \pi_k \mathrm{Normal}(x; \mu_k, \sigma_k^2) \\ q_{\phi}(x) =…

saved by

related reading