Things that confused me about cross-entropy · Chris Said

Every once in a while, I try to better understand cross-entropy by skimming over some Medium posts and StackExchange answers. I always come away only half-understanding it. A major cause of confusion is that different sources use different notations and conventions. Here are some that tripped me up. In information theory, the cross-entropy for an event with 𝑀 discrete outcome classes is … where 𝑝 is the true distribution of outcomes and 𝑞 is the approximating distribution. But in machine learning, the cross-entropy is defined as: … where 𝑦 are the true labels and 𝑝 ( 𝑦 ) is the approximating distribution, i.e. the classifier’s probability predictions. Notice that the meaning of 𝑝 has been reversed! In information theory, it is the true distribution. In machine learning, it is the approximating distribution. This points to a key difference between generic information theory and machine learning applications. In information theory, the target distribution 𝑝 is a pro

saved by

• Nick Dupoux