✳flâneur — a map of the web's best reading

A User’s Guide to FlexAttention in FlashAttention CuTe DSL – Colfax Research

research.colfax-intl.com · 3,050 words · saved by 1 readers

Many variants of attention (Vaswani et al., 2017) have become popular in recent years, for reasons related to performance and model quality. These include: The PyTorch team at Meta recognized that most of these variants (including all of the above) can be unified under one elegant framework, dubbed FlexAttention (Guessous et al., 2024). This simple API allows users to define and work with a large collection of attention variants, including novel combinations of existing ones, with relatively little development overhead and decent performance. FlexAttention adds two options for customization: a score_mod callable that modifies pre-softmax attention scores and a mask_mod callable that masks out pre-softmax attention scores. Altogether, FlexAttention takes the form Note that mask_mod is a special case of score_mod where scores are set to -inf; we keep the two separate for efficiency reasons, as will be explained when discussing block sparsity. The original FlexAttention implementation is

A User's Guide to FlexAttention in FlashAttention CuTe DSL - Colfax Research Skip to content A User’s Guide to FlexAttention in FlashAttention CuTe DSL Many variants of attention ( Vaswani et al., 2017 ) have become popular in recent years, for reasons related to performance and model quality. These include: Causal attention for autoregressive language modeling, where a token only attends to those prior; Sliding window attention for long-context language modeling, where a token only attends to those prior within a predefined window, reducing computational complexity of attention fro

Explore this link on the map →

saved by

related reading