Transformer (deep learning)

In deep learning, the transformer is a family of artificial neural network architectures based on the multi-head attention mechanism, in which text is converted to numerical representations called tokens, and each token is converted into a vector via lookup from a word embedding table. At each layer, each token is then contextualized within the scope of the context window with other (unmasked) tokens via a parallel multi-head attention mechanism, allowing the signal for key tokens to be amplified and less important tokens to be diminished. Because self-attention alone is permutation-invariant, transformers inject positional information, typically through positional encodings or learned positional embeddings, so token order can affect the output.

Transformer (deep learning) - Wikipedia Jump to content From Wikipedia, the free encyclopedia Algorithm for modelling sequential data This article has multiple issues. Please help improve it or discuss these issues on the talk page . ( Learn how and when to remove these messages ) This article needs additional citations for verification . Please help improve this article by adding citations to reliable sources . Unsourced material may be challenged and removed. Find sources:   "Transformer" deep learning  –  news   · newspapers   · books   · scholar   ·

saved by

• Lydia Nottingham

related reading

• Chapter 1: Transformer Interpretability - ARENAlearn.arena.education
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• Everything About Transformerskrupadave.com
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• 9 Transformers – 6.390 - Intro to Machine Learningintroml.mit.edu
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Annotated Transformernlp.seas.harvard.edu