flâneur

Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer | IEEE Conference Publication | IEEE Xplore

ieeexplore.ieee.org · 386 words · saved by 1 readers

Recurrent neural networks (RNNs), long-short memory (LSTM) [7], and gated recurrent (GRU) [3], used to be the best solutions in the natural language processing (NLP) area. This situation was changed when the Transformer model [11] was invented in 2017, which outperforms previous RNN models in multiple tasks. By avoiding the recurrent calculations and taking full advantage of the attention mechanism, the Transformer and Transformer-based pre-trained language models (such as BERT [4], ALBERT [8], T5 [9], ERINE [10], and structBERT [14]) have achieved state-of-the-art accuracy in various NLP tasks. In spite of making great progress in relative fields, the high computation complexity and huge memory requirements of these powerful Transformer networks are making them hard to be operated in mobile devices or embedded systems. More and more researchers are paying attention to this problem, and one way to solve it is through model compression [5]. Several techniques have been used to compress

Download PDF Download References Request Permissions Save to Alerts Abstract: Designing hardware accelerators for deep neural networks (DNNs) has been much desired. Nonetheless, most of these existing accelerators are built for either convolutional...Show More Metadata Abstract: Designing hardware accelerators for deep neural networks (DNNs) has been much desired. Nonetheless, most of these existing accelerators are built for either convolutional neural networks (CNNs) or recurrent neural networks (RNNs). Recently, the Transformer model is replacing the RNN in the natural language…

saved by

related reading