KL-divergence as an objective function — Graduate Descent
It's well-known that KL-divergence is not symmetric, but which direction is right for fitting your model? If we're fitting q θ 𝑞 𝜃 to p 𝑝 using KL(p|| q θ ) KL ( 𝑝 | | 𝑞 𝜃 ) mean-seeking, inclusive (more principled because approximates the full distribution) requires normalization wrt p 𝑝 (i.e., often not computationally convenient) KL( q θ ||p) KL ( 𝑞 𝜃 | | 𝑝 ) mode-seeking, exclusive no normalization wrt p 𝑝 (i.e., computationally convenient) Mnemonic: "When the truth comes first, you get the whole truth" (h/t Philip Resnik). Here "whole truth" corresponds to the inclusiveness of KL(p||q) KL ( 𝑝 | | 𝑞 ) . As far as remembering the equation, I pretend that " || | | " is a division symbol, which happens to correspond nicely to a division symbol in the equation (I'm not sure it's intentional). Let's look at what's involved in fitting a model q θ 𝑞 𝜃 in each direction. In this section, I'll describe the gradient and pay special attention to the issue of normaliz
Explore this link on the map →