Attention Sinks (Don't*) Play Topologically Important Roles in Current LLM Architecture | erkhesiin

TDA on Attention Sinks

December 21, 2025 Important Note It seems this analysis is only viable for the used model and prompt and does not generalize. Little frustrating, but it's fine. C'est la vie. Introduction The "Attention Sink" is the phenomenon where Large Language Models (LLMs) allocate a disproportionate amount of attention probability to the [BOS] token [1]. The consensus is that the sink is an artifact required for Softmax stability. Recent innovations in architecture, such as Gated Attention, show that the sink can be eliminated safely and doing so improves context utilization [1]. This raises the…

saved by

• Vihan Tiwari

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Circuit Tracing: Revealing Computational Graphs in Language Modelstransformer-circuits.pub
• In-context Learning and Induction Headstransformer-circuits.pub
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• Large Language Diffusion Modelsarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• A History of Large Language Modelsgregorygundersen.com
• 2502.11089arxiv.org
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencearxiv.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• [2602.05970] Inverse Depth Scaling From Most Layers Being Similararxiv.org
• How LLMs Actually Work | 0xkato0xkato.xyz