✳flâneur — a map of the web's best reading

Demystifying Sparse Attention: A Comprehensive Guide from Scratch | by VISHAL SINGH | Medium

medium.com · 1,137 words · saved by 1 readers

Transformers are powerful and versatile models for NLP, but they are limited by the memory and computation requirements of their full attention mechanism, which restricts us from increasing input sequence length. This makes them unsuitable for tasks that require longer contexts, such as question answering, document summarization, or genomic analysis. In this blog post, we learn about Big Bird, a new transformer model that can process sequences of up to 8 times longer than the standard transformer, using a sparse attention mechanism that reduces the computational and memory complexity from quadratic to linear. To delve into the mathematical understanding of attention layers, we need to break down the key components of the attention mechanism, particularly as it is applied in the context of neural networks like the Transformer. Let’s explore the mathematics behind the attention mechanism step by step. Here, in the above figure we have X_embedding, which is the embedding of our input sent

Demystifying Sparse Attention: A Comprehensive Guide from Scratch VISHAL SINGH 6 min read · Jan 14, 2024 -- 2 Listen Share Motivation : Transformers are powerful and versatile models for NLP, but they are limited by the memory and computation requirements of their full attention mechanism, which restricts us from increasing input sequence length. This makes them unsuitable for tasks that require longer contexts, such as question answering, document summarization, or genomic analysis. In this blog post, we learn about Big Bird, a new transformer model that can process sequences of up to 8 times

Explore this link on the map →

saved by

related reading