Luisa Shimabucoro
0 followers · 285 views
on the atlas · 3
- Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.9 savers
- The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.37 savers
- Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data Science1 savers
highlights · 11
The only disadvantage of the Attention mechanism is that it is a very time consuming and hard to parallelize system. To solve this problem, Google Brain came up with the “Transformer Model” which uses only Attention and gets rid of all the Convolutional and Recurrent Layers, thus making it highly parallelizable and compute efficient.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data Sciencethe model is able to find the correct local mappings between the input and the output sequences which do match with our intuition.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data ScienceNote that unlike the fixed context vector used for all the decoder time steps in case of the traditional Seq2Seq models, here in case of Attention, we compute a separate context vector for each time step by computing the attention weights every time.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data ScienceRecall that these states (h1 to h5) are nothing but vectors of fixed length. To develop some intuition think of these states as vectors which store local information within the sequence.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data ScienceThe central idea behind Attention is not to throw away those intermediate encoder states but to utilize all the states in order to construct the context vectors required by the decoder to generate the output sequence.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data ScienceA critical and apparent disadvantage of this fixed-length context vector design is the incapability of the system to remember longer sequences. Often is has forgotten the earlier parts of the sequence once it has processed the entire the sequence. The attention mechanism was born to resolve this problem.
Intuitive Understanding of Attention Mechanism in Deep Learning | by Harshall Lamba | Towards Data ScienceLet us now bring the whole thing together in the following visualization and look at how the attention process works: The attention decoder RNN takes in the embedding of the token, and an initial decoder hidden state. The RNN processes its inputs, producing an output and a new hidden state vector (h4). The output is discarded. Attention Step: We use the encoder hidden states and the h4 vector to calculate a context vector (C4) for this time step. We concatenate h4 and C4 into one vector. We pass this vector through a feedforward neural network (one trained jointly with the model). The output o…
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.The context vector turned out to be a bottleneck for these types of models. It made it challenging for the models to deal with long sentences. A solution was proposed in Bahdanau et al., 2014 and Luong et al., 2015. These papers introduced and refined a technique called “Attention”, which highly improved the quality of machine translation systems. Attention allows the model to focus on the relevant parts of the input sequence as needed.
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.Notice how the last hidden state is actually the context we pass along to the decoder.
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.To transform a word into a vector, we turn to the class of methods called “word embedding” algorithms. These turn words into vector spaces that capture a lot of the meaning/semantic information of the words
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.The encoder processes each item in the input sequence, it compiles the information it captures into a vector (called the context). After processing the entire input sequence, the encoder sends the context over to the decoder, which begins producing the output sequence item by item.
Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.