Your Transformer is Secretly an EOT Solver | Elements of a Vector Space

How the search for a simple approximation revealed a beautiful, exact solution.

Nine months ago, or was it ten? I've learned not to trust my sense of time when deep in a problem. Anyone faced with a major deadline or enveloped in the mire of one particular question for an extended period intuitively understands that Time becomes something else then. Not the measured progression of hours, but rather a kind of thickness, a medium through which one moves with increasing difficulty, like a swimmer who has ventured too far from shore and finds the water has become viscous, resistant. I was doing what I had seen others do, what perhaps I had been doing all along without quite a

saved by

• Kushal Thaman
• Vallabh
• Vihan Tiwari
• Ishaan Panigrahi

related reading

• The Annotated Transformernlp.seas.harvard.edu
• A Theory of Deep Learning | Elements of a Vector Spaceelonlit.com
• transformer_attention.pdfarxiv.org
• Transformers Laid Outgoyalpramod.github.io
• Attention Is Off By One – Evan Millerevanmiller.org
• Transformers from Scratche2eml.school
• [2205.14135] FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenessarxiv.org
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io
• An Intuition for Attention | Jay Modyjaykmody.com
• Elon Litman | Elements of a Vector Spaceelonlit.com
• The Annotated Transformernlp.seas.harvard.edu
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai