Learning to (Learn at Test Time): RNNs with Expressive Hidden States

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on. Authors: achieve the best HTML results from your LaTeX submissions by following these best practices. Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidde

Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun 1 1 ~\,{}^{1} start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT , Xinhao Li ∗ 2 , Karan Dalal ∗ 3 , Jiarui Xu 2 , Arjun Vikram 1 , Genghan Zhang 1 , Yann Dubois 1 , Xinlei Chen † 4 , Xiaolong Wang † 2 , Sanmi Koyejo † 1 , Tatsunori Hashimoto † 1 , Carlos Guestrin † 1 1 Stanford University 2 UC San Diego 3 UC Berkeley 4 Meta AI Core contributors. † Joint advising. See author contributions at the end of the paper. Correspondence to: ys646@stanford.edu , xil202@ucsd.edu , kdalal@berkeley.edu . Code available in JAX and PyTorc

related reading

• [2407.04620] Learning to (Learn at Test Time): RNNs with Expressive Hidden Statesar5iv.labs.arxiv.org
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Test-Time Training Done Rightarxiv.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Little Book of Deep Learningfleuret.org
• microgptkarpathy.github.io
• NL.pdfabehrouz.github.io
• Tiny TPUtinytpu.com
• DeltaNet Explained (Part I) | Songlin Yangsustcsonglin.github.io
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org