Attention and Augmented Recurrent Neural Networks

A visual overview of neural attention, and the powerful extensions of neural networks being built on top of it.

Recurrent neural networks are one of the staples of deep learning, allowing neural networks to work with sequences of data like text, audio and video. They can be used to boil a sequence down into a high-level understanding, to annotate sequences, and even to generate new sequences from scratch! rnn x0 y0 x1 y1 x2 y2 x3 y3 One cell... can be used over... and over... and over... x4 y4 again. The basic RNN design struggles with longer sequences, but a special variant—“long short-term memory” networks [1]—can even work with these. Such models have been found to be very powerful,…

saved by

• Donald

related reading

• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Ilya 30u30arc.net
• transformer_attention.pdfarxiv.org
• Understanding LSTM Networks -- colah's blogcolah.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• Recurrent neural network - Wikipediaen.wikipedia.org
• Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs · Denny's Blogdennybritz.com
• Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• 1603.08983arxiv.org
• 1706.03762arxiv.org