✳flâneur — a map of the web's best reading

Transformers without Normalization - DynamicTanh - DyT

jiachenzhu.github.io · 509 words · saved by 1 readers

Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh (DyT), an element-wise operation DyT ( \boldsymbol 𝑥 ) = tanh ⁡ ( 𝛼 \boldsymbol 𝑥 ) , as a drop-in replacement for normalization layers in Transformers. DyT is inspired by the observation that layer normalization in Transformers often produces tanh-like, S-shaped input-output mappings. By incorporating DyT, Transformers without normalization can match or exceed the performance of their normalized counterparts, mostly without hyperparameter tuning. We validate the effectiveness of Transformers with DyT across diverse settings, ranging from recognition to generation, supervised to self-supervised learning, and computer vision to language models. These findings challenge the conventional understanding that normalizatio

Transformers without Normalization - DynamicTanh - DyT Left: original Transformer block. Right: block with our proposed Dynamic Tanh (DyT) layer. DyT is a straightforward replacement for commonly used Layer Norm or RMSNorm layers. Transformers with DyT match or exceed the performance of their normalized counterparts. Abstract Normalization layers are ubiquitous in modern neural networks and have long been considered essential. This work demonstrates that Transformers without normalization can achieve the same or better performance using a remarkably simple technique. We introduce Dynamic Tanh

Explore this link on the map →

related reading