Fabian Fuchs
# Gumbel Softmax This post introduces the Gumbel Softmax estimator for stochastic neural networks. It was simultaneously discovered by [Maddison et al.](https://arxiv.org/abs/1611.00712) and [Jang et al.](https://arxiv.org/abs/1611.01144), with both papers published at ICLR 2017. I took figures, equations and notation from the latter of the two. ## Vanilla Gumbel Softmax Estimator...
Fabian Fuchs Gumbel Softmax This post introduces the Gumbel Softmax estimator for stochastic neural networks. It was simultaneously discovered by Maddison et al. and Jang et al. , with both papers published at ICLR 2017. I took figures, equations and notation from the latter of the two. Vanilla Gumbel Softmax Estimator The Gumbel Softmax trick can be looked at from different angles. I will approach it from an attention angle, which has a broad range of applications in deep learning. For example, imagine a neural network that processes an image and decomposes it into multiple objects. Now we wa
saved by
related reading
- Generative Modeling by Estimating Gradients of the Data Distribution | Yang Songyang-song.github.io
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation | Yang Songyang-song.github.io
- [1611.01144] Categorical Reparameterization with Gumbel-Softmaxarxiv.org
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- H-Nets - the Past | Goomba Labgoombalab.github.io
- [1802.08665] Learning Latent Permutations with Gumbel-Sinkhorn Networksarxiv-vanity.com
- Understanding Convolutions on Graphsdistill.pub
- [2605.01172] A Theory of Generalization in Deep Learningarxiv.org
- [1706.04599] On Calibration of Modern Neural Networksarxiv.org
- Understanding deep learning requires rethinking generalizationarxiv.org
- [1809.09087] Implicit Maximum Likelihood Estimationarxiv.org
- From Online Softmax to FlashAttentioncourses.cs.washington.edu