flâneur

Transformer: A Novel Neural Network Architecture for Language Understanding – Google Research Blog

blog.research.google · 1,344 words · saved by 1 readers

Neural networks, in particular recurrent neural networks (RNNs), are now at the core of the leading approaches to language understanding tasks such as language modeling, machine translation and question answering. In “Attention Is All You Need”, we introduce the Transformer, a novel neural network architecture based on a self-attention mechanism that we believe to be particularly well suited for language understanding. In our paper, we show that the Transformer outperforms both recurrent and convolutional models on academic English to German and English to French translation benchmarks. On top of higher translation quality, the Transformer requires less computation to train and is a much better fit for modern machine learning hardware, speeding up training by up to an order of magnitude. Neural networks usually process language by generating fixed- or variable-length vector-space representations. After starting with representations of individual words or even pieces of words, they aggr

Transformer: A Novel Neural Network Architecture for Language Understanding Skip to main content Transformer: A Novel Neural Network Architecture for Language Understanding August 31, 2017 Posted by Jakob Uszkoreit, Software Engineer, Natural Language Understanding Quick links Share Copy link × Neural networks, in particular recurrent neural networks (RNNs), are now at the core of the leading approaches to language understanding tasks such as language modeling , machine translation and question answering . In “ Attention Is All You Need ”, we introduce the Transformer, a novel neural network a

related reading