✳flâneur — a map of the web's best reading

Learning to (Learn at Test Time): RNNs with Expressive Hidden States | HTML5

ar5iv.labs.arxiv.org · 21,065 words · saved by 1 readers

Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Both TTT-Linear and TTT-MLP match or exceed the baselines. Similar to Transformer, they can keep reducing perplexity by conditioning on more tokens, while Mamba can

Learning to (Learn at Test Time): RNNs with Expressive Hidden States Yu Sun 1 1 ~\,{}^{1} , Xinhao Li ∗ 2 , Karan Dalal ∗ 3 , Jiarui Xu 2 , Arjun Vikram 1 , Genghan Zhang 1 , Yann Dubois 1 , Xinlei Chen † 4 , Xiaolong Wang † 2 , Sanmi Koyejo † 1 , Tatsunori Hashimoto † 1 , Carlos Guestrin † 1 Core contributors. † Joint advising. See author contributions at the end of the paper. 1 Stanford University. 2 UC San Diego. 3 UC Berkeley. 4 Meta AI. Correspondence to: yusun@cs.stanford.edu , xil202@ucsd.edu , kdalal@berkeley.edu . Code available in JAX and PyTorch . Abstract Self-attention performs we

Explore this link on the map →

related reading