[1706.03762] Attention Is All You Need

Abstract:The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles by over 2 BLEU. On the WMT 2014 English-to-French translation task, our model establishes a new single-model state-of-the-art BLEU score of 41.8 after training for 3.5 days on eight GPUs, a small fraction of the training costs of the best models from the literature. We show that the Transformer generalizes well to other tasks by applying it successfully to English constituency parsing both with large and limited training data.

[1706.03762] Attention Is All You Need --> Computer Science > Computation and Language arXiv:1706.03762 (cs) [Submitted on 12 Jun 2017 ( v1 ), last revised 2 Aug 2023 (this version, v7)] Title: Attention Is All You Need Authors: Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan N. Gomez , Lukasz Kaiser , Illia Polosukhin View a PDF of the paper titled Attention Is All You Need, by Ashish Vaswani and 7 other authors View PDF HTML (experimental) Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in

saved by

• Krish Shah
• Elizabeth Qiu
• Jules Qian
• Elliot Kim
• David A. Lee
• Wendi Zheng
• Lena Araujo
• Emily Qi
• 591

related reading

• transformer_attention.pdfarxiv.org
• 1706.03762arxiv.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformers from Scratche2eml.school
• Seq2seq and Attentionlena-voita.github.io
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer: A Novel Neural Network Architecture for Language Understandingblog.research.google
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Attention is all you need: Discovering the Transformer paper | Towards Data Sciencetowardsdatascience.com
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• The Annotated Transformernlp.seas.harvard.edu