[2103.03386] Clusterability in Neural Networks

The learned weights of a neural network have often been considered devoid of scrutable internal structure. In this paper, however, we look for structure in the form of clusterability: how well a network can be divided into groups of neurons with strong internal connectivity but weak external connectivity. We find that a trained neural network is typically more clusterable than randomly initialized networks, and often clusterable relative to random networks with the same distribution of weights. We also exhibit novel methods to promote clusterability in neural network training, and find that in multi-layer perceptrons they lead to more clusterable networks with little reduction in accuracy. Understanding and controlling the clusterability of neural networks will hopefully render their inner workings more interpretable to engineers by facilitating partitioning into meaningful clusters.

Clusterability in Neural Networks Daniel Filan1, * , Stephen Casper2, * , Shlomi Hod3, * , Cody Wild1 , Andrew Critch1 , Stuart Russell1 1 UC Berkeley, 2 Harvard, 3 Boston University {daniel filan, codywild, critch,…

related reading

• Toy Models of Superpositiontransformer-circuits.pub
• Neural Networks, Manifolds, and Topology -- colah's blogcolah.github.io
• The Building Blocks of Interpretabilitydistill.pub
• [1706.04599] On Calibration of Modern Neural Networksarxiv.org
• [1911.08731] Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationarxiv.org
• Understanding deep learning requires rethinking generalizationarxiv.org
• Multimodal Neurons in Artificial Neural Networksdistill.pub
• NL.pdfabehrouz.github.io
• Feature Visualizationdistill.pub
• [2604.21691] There Will Be a Scientific Theory of Deep Learningarxiv.org
• Naturally Occurring Equivariance in Neural Networksdistill.pub
• On_the_Spectral_Bias_of_Neural_Networks__ICML_ (3)proceedings.mlr.press