flâneur

The information bottleneck method | HTML5

ar5iv.labs.arxiv.org · 8,330 words · saved by 1 readers

We define the relevant information in a signal 𝑥 ∈ 𝑋 as being the information that this signal provides about another signal 𝑦 ∈ 𝑌 . Examples include the information that face images provide about the names of the people portrayed, or the information that speech sounds provide about the words spoken. Understanding the signal 𝑥 requires more than just predicting 𝑦 , it also requires specifying which features of 𝑋 play a role in the prediction. We formalize this problem as that of finding a short code for 𝑋 that preserves the maximum information about 𝑌 . That is, we squeeze the information that 𝑋 provides about 𝑌 through a ‘bottleneck’ formed by a limited set of codewords 𝑋 ~ . This constrained optimization problem can be seen as a generalization of rate distortion theory in which the distortion measure 𝑑 ​ ( 𝑥 , 𝑥 ~ ) emerges from the joint statistics of 𝑋 and 𝑌 . This approach yields an exact set of self consistent equations for the coding rules

The information bottleneck method Naftali Tishby, 1,2 Fernando C. Pereira, 3 and William Bialek 1 1 NEC Research Institute, 4 Independence Way Princeton, New Jersey 08540 2 Institute for Computer Science, and Center for Neural Computation Hebrew University Jerusalem 91904, Israel 3 AT&T Shannon Laboratory 180 Park Avenue Florham Park, New Jersey 07932 30 September 1999 We define the relevant information in a signal x ∈ X 𝑥 𝑋 x\in X as being the information that this signal provides about another signal y ∈ Y 𝑦 𝑌 y\in{Y} . Examples include the information that face images provide about the

related reading