Attention? Attention! | Lil'Log

[Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.] [Updated on 2018-11-06: Add a link to the implementation of Transformer model.] [Updated on 2018-11-18: Add Neural Turing Machines.] [Updated on 2019-07-18: Correct the mistake on using the term “self-attention” when introducing the show-attention-tell paper; moved it to Self-Attention section.] [Updated on 2020-04-07: A follow-up post on improved Transformer models is here.] Attention is, to some extent, motivated by how we pay visual attention to different regions of an image or correlate words in one sentence.

Table of Contents What’s Wrong with Seq2Seq Model? Born for Translation Definition A Family of Attention Mechanisms Summary Self-Attention Soft vs Hard Attention Global vs Local Attention Neural Turing Machines Reading and Writing Attention Mechanisms Pointer Network Transformer Key, Value and Query Multi-Head Self-Attention Encoder Decoder Full Architecture SNAIL Self-Attention GAN References [Updated on 2018-10-28: Add Pointer Network and the link to my implementation of Transformer.] [Updated on 2018-11-06: Add a link to the implementation of Transformer model.] [Updated on 2018-11-18: Add

related reading

• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• transformer_attention.pdfarxiv.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 1706.03762arxiv.org
• Everything About Transformerskrupadave.com
• Seq2seq and Attentionlena-voita.github.io
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Attention is all you need: Discovering the Transformer paper | Towards Data Sciencetowardsdatascience.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• What is an attention mechanism? | IBMibm.com
• Transformers from scratch | peterbloem.nlpeterbloem.nl
• Transformers from Scratche2eml.school