[2601.12703] Towards Spectroscopy: Susceptibility Clusters in Language Models
Abstract:Spectroscopy infers the internal structure of physical systems by measuring their response to perturbations. We apply this principle to neural networks: perturbing the data distribution by upweighting a token $y$ in context $x$, we measure the model's response via susceptibilities $\chi_{xy}$, which are covariances between component-level observables and the perturbation computed over a localized Gibbs posterior via stochastic gradient Langevin dynamics (SGLD). Theoretically, we show that susceptibilities decompose as a sum over modes of the data distribution, explaining why tokens that follow their contexts "for similar reasons" cluster together in susceptibility space. Empirically, we apply this methodology to Pythia-14M, developing a conductance-based clustering algorithm that identifies 510 interpretable clusters ranging from grammatical patterns to code structure to mathematical notation. Comparing to sparse autoencoders, 50% of our clusters match SAE features, validating that both methods recover similar structure.
Towards Spectroscopy: Susceptibility Clusters in Language Models Andrew Gordon= Garrett Baker= George Wang Timaeus Timaeus Timaeus andrew@timaeus.co garrett@timaeus.co george@timaeus.co arXiv:2601.12703v1 [cs.LG] 19 Jan 2026…
saved by
related reading
- Towards Automated Circuit Discovery for Mechanistic Interpretabilityarxiv.org
- [2607.14111] Introspection Fine-Tuning (IFT): Training Small LLMs to Introspectarxiv.org
- Training Language Models to Explain Their Own Computationsarxiv.org
- [2511.08579] Training Language Models to Explain Their Own Computationsarxiv.org
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
- [2210.13382] Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskarxiv.org
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
- On the Biology of a Large Language Modeltransformer-circuits.pub
- [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org