Modeling Symmetries | Mark Neumann
Chaitanya K. Joshi's excellent blog post on his PhD work inspired me to write down some thoughts on symmetries and invariances in neural network models—particularly in the context of molecular data. Discussions around removing architectural biases can lean a little absolutist ("Bitter Lesson, GPUs go brrr!"), whereas the reality is more nuanced. I am actually not so opposed to the idea of encoding physical priors, but I am interested in when such symmetries should be embedded directly into the model architecture, and when they might be better handled implicitly: through data augmentation, regularization or inference-time tricks. This post represents some of my current thinking on the topic, through the lens of two modeling domains in which the approaches to encoding symmetries are very different: neural network potentials and diffusion models. When modeling image data, there are many desirable invariances one can imagine (translation being the predominant one baked into the most common
Chaitanya K. Joshi's excellent blog post on his PhD work inspired me to write down some thoughts on symmetries and invariances in neural network models—particularly in the context of molecular data. Discussions around removing architectural biases can lean a little absolutist (" Bitter Lesson , GPUs go brrr!"), whereas the reality is more nuanced. I am actually not so opposed to the idea of encoding physical priors, but I am interested in when such symmetries should be embedded directly into the model architecture, and when they might be better handled implicitly: through data augmentation, re
saved by
related reading
- A Recipe for Training Neural Networkskarpathy.github.io
- What are Diffusion Models? | Lil'Loglilianweng.github.io
- Naturally Occurring Equivariance in Neural Networksdistill.pub
- Toy Models of Superpositiontransformer-circuits.pub
- Modular Manifolds - Thinking Machines Labthinkingmachines.ai
- 10. Equivariant Neural Networks — deep learning for molecules & materialsdmol.pub
- Beyond Message Passing: a Physics-Inspired Paradigm for Graph Neural Networksthegradient.pub
- Statistical Mechanics of Deep Learningganguli-gang.stanford.edu
- 1802.08219arxiv.org
- On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
- A Recipe for Training Neural Networkskarpathy.github.io
- [2203.17003] Equivariant Diffusion for Molecule Generation in 3Darxiv.org