flâneur

Things that confused me about cross-entropy · Chris Said

chris-said.io · 604 words · saved by 1 readers

Every once in a while, I try to better understand cross-entropy by skimming over some Medium posts and StackExchange answers. I always come away only half-understanding it. A major cause of confusion is that different sources use different notations and conventions. Here are some that tripped me up. In information theory, the cross-entropy for an event with 𝑀 discrete outcome classes is … where 𝑝 is the true distribution of outcomes and 𝑞 is the approximating distribution. But in machine learning, the cross-entropy is defined as: … where 𝑦 are the true labels and 𝑝 ( 𝑦 ) is the approximating distribution, i.e. the classifier’s probability predictions. Notice that the meaning of 𝑝 has been reversed! In information theory, it is the true distribution. In machine learning, it is the approximating distribution. This points to a key difference between generic information theory and machine learning applications. In information theory, the target distribution 𝑝 is a pro

26 Dec 2020 Every once in a while, I try to better understand cross-entropy by skimming over some Medium posts and StackExchange answers. I always come away only half-understanding it. A major cause of confusion is that different sources use different notations and conventions. Here are some that tripped me up. p and q vs y and p In information theory, the cross-entropy for an event with \(M\) discrete outcome classes is \[H(p, q) = -\sum_{j=1}^M{p_j log (q_j)}\] … where \(p\) is the true distribution of outcomes and \(q\) is the approximating distribution. But in machine learning,…

saved by

related reading