✳flâneur — a map of the web's best reading

Deep Variational Information Bottleneck | HTML5

ar5iv.labs.arxiv.org · 15,220 words · saved by 1 readers

We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method “Deep Variational Information Bottleneck”, or Deep VIB. We show that models trained with the VIB objective outperform those that are trained with other forms of regularization, in terms of generalization performance and robustness to adversarial attack. We adopt an information theoretic view of deep networks. We regard the internal representation of some intermediate layer as a stochastic encoding 𝑍 of the input source 𝑋 , defined by a parametric encoder 𝑝 ​ ( 𝐳 | 𝐱 ; 𝜽 ) .1 1 In this work, 𝑋 , 𝑌 , 𝑍 are random variables, 𝑥 , 𝑦 , 𝑧 and 𝐱 , 𝐲 , 𝐳 are instances of random variables, and 𝐹 ​ ( ⋅ ; 𝜽 ) and 𝑓 ​ ( ⋅ ; 𝜽 ) are functionals or functions parameterized by 𝜽 . Our

Deep Variational information bottleneck Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy Google Research {alemi,iansf,jvdillon,kpmurphy}@google.com Abstract We present a variational approximation to the information bottleneck of Tishby et al. ( 1999 ) . This variational approach allows us to parameterize the information bottleneck model using a neural network and leverage the reparameterization trick for efficient training. We call this method “Deep Variational Information Bottleneck”, or Deep VIB. We show that models trained with the VIB objective outperform those that are trai

Explore this link on the map →

related reading