Attention Sinks (Don't*) Play Topologically Important Roles in Current LLM Architecture | erkhesiin

TDA on Attention Sinks

December 21, 2025 Important Note It seems this analysis is only viable for the used model and prompt and does not generalize. Little frustrating, but it's fine. C'est la vie. Introduction The "Attention Sink" is the phenomenon where Large Language Models (LLMs) allocate a disproportionate amount of attention probability to the [BOS] token [1]. The consensus is that the sink is an artifact required for Softmax stability. Recent innovations in architecture, such as Gated Attention, show that the sink can be eliminated safely and doing so improves context utilization [1]. This raises the…

saved by

• Vihan Tiwari

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• How LLMs Actually Work | 0xkato0xkato.xyz
• The Big LLM Architecture Comparisonmagazine.sebastianraschka.com
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Understanding Attention in LLMs | Bartosz Milewski's Programming Cafebartoszmilewski.com
• Attention Is Off By One – Evan Millerevanmiller.org
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
• [2402.09221] Spectral Filters, Dark Signals, and Attention Sinksarxiv.org
• Some Intuition on Attention and the Transformereugeneyan.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Extending Context is Hard | kaiokendevkaiokendev.github.io