Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer | IEEE Conference Publication | IEEE Xplore

Recurrent neural networks (RNNs), long-short memory (LSTM) [7], and gated recurrent (GRU) [3], used to be the best solutions in the natural language processing (NLP) area. This situation was changed when the Transformer model [11] was invented in 2017, which outperforms previous RNN models in multiple tasks. By avoiding the recurrent calculations and taking full advantage of the attention mechanism, the Transformer and Transformer-based pre-trained language models (such as BERT [4], ALBERT [8], T5 [9], ERINE [10], and structBERT [14]) have achieved state-of-the-art accuracy in various NLP tasks. In spite of making great progress in relative fields, the high computation complexity and huge memory requirements of these powerful Transformer networks are making them hard to be operated in mobile devices or embedded systems. More and more researchers are paying attention to this problem, and one way to solve it is through model compression [5]. Several techniques have been used to compress

saved by

• Pavel