[2410.14687] BrainTransformers: SNN-LLM
Abstract:This study introduces BrainTransformers, an innovative Large Language Model (LLM) implemented using Spiking Neural Networks (SNN). Our key contributions include: (1) designing SNN-compatible Transformer components such as SNNMatmul, SNNSoftmax, and SNNSiLU; (2) implementing an SNN approximation of the SiLU activation function; and (3) developing a Synapsis module to simulate synaptic plasticity. Our 3-billion parameter model, BrainTransformers-3B-Chat, demonstrates competitive performance across various benchmarks, including MMLU (63.2), BBH (54.1), ARC-C (54.3), and GSM8K (76.3), while potentially offering improved energy efficiency and biological plausibility. The model employs a three-stage training approach, including SNN-specific neuronal synaptic plasticity training. This research opens new avenues for brain-like AI systems in natural language processing and neuromorphic computing. Future work will focus on hardware optimization, developing specialized SNN fine-tuning tools, and exploring practical applications in energy-efficient computing environments.
View PDF HTML (experimental) Abstract:This study introduces BrainTransformers, an innovative Large Language Model (LLM) implemented using Spiking Neural Networks (SNN). Our key contributions include: (1) designing SNN-compatible Transformer components such as SNNMatmul, SNNSoftmax, and SNNSiLU; (2) implementing an SNN approximation of the SiLU activation function; and (3) developing a Synapsis module to simulate synaptic plasticity. Our 3-billion parameter model, BrainTransformers-3B-Chat, demonstrates competitive performance across various benchmarks, including MMLU (63.2), BBH (54.1),…
saved by
related reading
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- [2302.13939] SpikeGPT: Generative Pre-trained Language Model with Spiking Neural Networksarxiv.org
- transformer_attention.pdfarxiv.org
- Language Model Circuits Are Sparse in the Neuron Basis | Transluce AItransluce.org
- Brainformers: Trading Simplicity for Efficiencyarxiv.org
- Scalable MatMul-free Language Modelingarxiv.org
- Large language models surpass human experts in predicting neuroscience resultsnature.com
- Exploring spiking neural networks: a comprehensive analysis of mathematical models and applicationsfrontiersin.org
- Softmax Linear Unitstransformer-circuits.pub
- Language models can explain neurons in language modelsopenaipublic.blob.core.windows.net
- HumanLM: Simulating Users with State Alignment Beats Response Imitationarxiv.org
- Towards a “universal translator” for neural dynamics at single-cell, single-spike resolutionarxiv.org