GPT-2's positional embedding matrix is a helix — LessWrong

In the context of transformer models, the "positional embedding matrix" is the thing that encodes the meaning of positions within a prompt. For example, given the prompt: Hello my name is Adam the prompt would generally be broken down into tokens as follows: [' ', 'Hello', ' my', ' name', ' is', ' Adam'] (For whatever reason prompts to GPT-2 generally have an token prepended to them before being fed through, to match how the model was trained.) For this prompt, the mapping of tokens to positions would be as follows: ' ': 0 'Hello': 1 ' my': 2 ' name': 3 ' is': 4 ' Adam': 5 The positional embedding maps the positions of those tokens (0, 1, 2, 3, 4, and 5) to the meanings of those positions in vectorspace. More concretely, the positional embedding matrix maps each of those five numbers to a 768-dimensional vector of floating-point numbers, and that 768-dimensional vector gets added to a different vector that represents the semantic meaning of the tok

x GPT-2's positional embedding matrix is a helix — LessWrong GPT Interpretability (ML & AI) Machine Learning (ML) Transformers AI Frontpage 52 GPT-2's positional embedding matrix is a helix by AdamYedidia 21st Jul 2023 5 min read 21 52 In the context of transformer models, the "positional embedding matrix" is the thing that encodes the meaning of positions within a prompt. For example, given the prompt: Hello my name is Adam the prompt would generally be broken down into tokens as follows: ['<|endoftext|>', 'Hello', ' my', ' name', ' is', ' Adam'] (For whatever reason prompts to GPT-2 generall

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Master Positional Encoding: Part I | Towards Data Sciencetowardsdatascience.com
• Transformers from Scratche2eml.school
• Tiny Mech Interp Projects: Emergent Positional Embeddings of Words - Neel Nandaneelnanda.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformer Architecture: The Positional Encoding - Amirhossein Kazemnejad's Blogkazemnejad.com
• A Conceptual Guide to Transformers: Part Ibenlevinstein.substack.com
• interpreting GPT: the logit lens — AI Alignment Forumalignmentforum.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• interpreting GPT: the logit lens — LessWronglesswrong.com
• On N-dimensional Rotary Positional Embeddingsjerryxio.ng