“Soft” Dictionary Keys

Transformer Neural Networks utilize the key-concept of an Attention Mechanism to perform “lookups” on the data it has seen. In this post I want to detail the idea of “soft” keys, and for me it was easier to get to the crux of how Transformers work with this understanding. I first came across this idea from a Lucas Beyer talk. Most programming languages implement a dictionary (or associative map) as a primitive data structure and define them as associations between the abstract idea of keys and values. In python keys are defined as any hashable object. For example, Here, we have four keys, "dog", "cat", "tiger", 8 and they are mapped to values. The first three keys are Strings and the fourth key is a Number (integer in this case). All the values here are Numbers as well. Internally, python calls the in-built hash method to hash the keys into a well-known or fixed representation, The key point here is that keys are converted into a well-defined representation. In the case of python the r

“Soft” Dictionary Keys Transformer Neural Networks utilize the key-concept of an Attention Mechanism to perform “lookups” on the data it has seen. In this post I want to detail the idea of “soft” keys, and for me it was easier to get to the crux of how Transformers work with this understanding. I first came across this idea from a Lucas Beyer talk . Python Dictionaries Most programming languages implement a dictionary (or associative map) as a primitive data structure and define them as associations between the abstract idea of keys and values. In python keys are defined as any hashable object

saved by

• Raayan Dhar

related reading

• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Transformers from Scratche2eml.school
• An Intuition for Attention | Jay Modyjaykmody.com
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• neural networks - What exactly are keys, queries, and values in attention mechanisms? - Cross Validatedstats.stackexchange.com
• Ask HN: Can someone ELI5 transformers and the “Attention is all we need” paper? | Hacker Newsnews.ycombinator.com
• Attention Is Off By One – Evan Millerevanmiller.org
• Towards Monosemanticity: Decomposing Language Models With Dictionary Learningtransformer-circuits.pub
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transformers from Scratchbrandonrohrer.com