A Geometric Calculator Inside a Neural Network
We found a neural mechanism that operates over manifolds: a general-purpose addition module inside Llama 3.1 8B which manipulates circular representations of numbers.
A Geometric Calculator Inside a Neural Network Research ← The Neural Geometry Series → A Geometric Calculator Inside a Neural Network We found a neural mechanism that operates over manifolds: a general-purpose addition module inside Llama 3.1 8B which manipulates circular representations of numbers. Authors Sheridan Feucht *,1,2 Ekdeep Singh Lubana †,1 Tal Haklay *,1,3 Thomas Fel †,1 Usha Bhalla 1,4 Atticus Geiger †,1 Daniel Wurgaft 1,5 Can Rager 1 * Equal contribution Raphaël Sarfati 1 † Equal senior contribution Jack Merullo 1 1 Goodfire Thomas McGrath 1 2 Northeastern University Owen Lewis
Explore this link on the map →saved by
related reading
- The World Inside Neural Networksgoodfire.ai
- On the Biology of a Large Language Modeltransformer-circuits.pub
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- Zoom In: An Introduction to Circuitsdistill.pub
- Tracing the thoughts of a large language model \ Anthropicanthropic.com
- Transformer Circuits Threadtransformer-circuits.pub
- A Mechanistic Interpretability Analysis of Grokking — AI Alignment Forumalignmentforum.org
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- Steering Along Manifolds to Control Neural Networksgoodfire.ai
- Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activationstransformer-circuits.pub
- Neuronpedianeuronpedia.org
- [2602.16849] On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokkingarxiv.org