Rindra Ramamonjison
0 followers · 271 views
on the atlas · 2
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.37 savers
- Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Medium1 savers
highlights · 34
The sixth step is to sum up the weighted value vectors.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The fifth step is to multiply each value vector by the softmax score (in preparation to sum them up). The intuition here is to keep intact the values of the word(s) we want to focus on, and drown-out irrelevant words
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.then pass the result through a softmax operation. Softmax normalizes the scores so they’re all positive and add up to 1.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The third and fourth steps are to divide the scores by 8 (the square root of the dimension of the key vectors used in the paper – 64.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The score determines how much focus to place on other parts of the input sentence as we encode a word at a certain position.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The second step in calculating self-attention is to calculate a score.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.What are the “query”, “key”, and “value” vectors?
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Multiplying x1 by the WQ weight matrix produces q1, the "query" vector associated with that word. We end up creating a "query", a "key", and a "value" projection of each word in the input sentence.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.These vectors are created by multiplying the embedding by three matrices that we trained during the training process
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.for each word, we create a Query vector, a Key vector, and a Value vector
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The first step in calculating self-attention
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Self-Attention in Detail
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.self attention allows it to look at other positions in the input sequence for clues that can help lead to a better encoding for this word
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.When the model is processing the word “it”, self-attention allows it to associate “it” with “animal”.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Say the following sentence is an input sentence we want to translate: ”The animal didn't cross the street because it was too tired”
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Self-Attention at a High Level
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.As we’ve mentioned already, an encoder receives a list of vectors as input. It processes this list by passing these vectors into a ‘self-attention’ layer, then into a feed-forward neural network, then sends out the output upwards to the next encoder.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The feed-forward layer does not have those dependencies, however, and thus the various paths can be executed in parallel while flowing through the feed-forward layer.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.There are dependencies between these paths in the self-attention layer.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.the word in each position flows through its own path in the encoder
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.After embedding the words in our input sequence, each of them flows through each of the two layers of the encoder.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The embedding only happens in the bottom-most encoder.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.As is the case in NLP applications in general, we begin by turning each input word into a vector using an embedding algorithm.
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The decoder has both those layers, but between them is an attention layer that helps the decoder focus on relevant parts of the input sentence
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The outputs of the self-attention layer are fed to a feed-forward neural network. The exact same feed-forward network is independently applied to each position
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.The encoder’s inputs first flow through a self-attention layer – a layer that helps the encoder look at other words in the input sentence as it encodes a specific word
The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.Word Embedding aims to create a vector representation with a much lower dimensional space. These are called Word Vectors
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediumis equal to the size of the supported vocabulary
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediumone-hot encoded vectors
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediumexpressive representation
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Mediumefficient representation
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | MediumWord Embedding is to turn text into numbers
Glossary of Deep Learning: Word Embedding | by Jaron Collis | Deeper Learning | Medium