A Geometric Calculator Inside a Neural Network
We found a neural mechanism that operates over manifolds: a general-purpose addition module inside Llama 3.1 8B which manipulates circular representations of numbers.
A Geometric Calculator Inside a Neural Network Research ← The Neural Geometry Series → A Geometric Calculator Inside a Neural Network We found a neural mechanism that operates over manifolds: a general-purpose addition module inside Llama 3.1 8B which manipulates circular representations of numbers. Authors Sheridan Feucht *,1,2 Ekdeep Singh Lubana †,1 Tal Haklay *,1,3 Thomas Fel †,1 Usha Bhalla 1,4 Atticus Geiger †,1 Daniel Wurgaft 1,5 Can Rager 1 * Equal contribution Raphaël Sarfati 1 † Equal senior contribution Jack Merullo 1 1 Goodfire Thomas McGrath 1 2 Northeastern University Owen Lewis
saved by
related reading
- The World Inside Neural Networksgoodfire.ai
- The Neural Geometry Series - Goodfiregoodfire.com
- Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- NL.pdfabehrouz.github.io
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
- On the Biology of a Large Language Modeltransformer-circuits.pub
- Verbalizable Representations Form a Global Workspace in Language Modelstransformer-circuits.pub
- Symmetry in language statistics shapes the geometry of model representationsarxiv.org
- Steering Along Manifolds to Control Neural Networksgoodfire.ai
- Transformer Circuits Threadtransformer-circuits.pub
- [2602.15029] Symmetry in language statistics shapes the geometry of model representationsarxiv.org