Test-time regression: a unifying framework for designing sequence models with associative memory | HTML5

ar5iv.labs.arxiv.org · 16,096 words · saved by 1 readers

Sequences provide a remarkably general way to represent and process information. This powerful abstraction has placed sequence modeling at the center of modern deep learning applications, inspiring numerous architectures from transformers to recurrent networks. While this fragmented development has yielded powerful models, it has left us without a unified framework to understand their fundamental similarities and explain their effectiveness. We present a unifying framework motivated by an empirical observation: effective sequence models must be able to perform associative recall. Our key insight is that memorizing input tokens through an associative memory is equivalent to performing regression at test-time. This regression-memory correspondence provides a framework for deriving sequence models that can perform associative recall, offering a systematic lens to understand seemingly ad-hoc architectural choices. We show numerous recent architectures — including linear attention models, t

Test-time regression: a unifying framework for designing sequence models with associative memory Ke Alexander Wang 1 1 1 Correspondence to alxwang@cs.stanford.edu Jiaxin Shi 2 2 2 Now at Google Deepmind Emily B. Fox Abstract Sequences provide a remarkably general way to represent and process information. This powerful abstraction has placed sequence modeling at the center of modern deep learning applications, inspiring numerous architectures from transformers to recurrent networks. While this fragmented development has yielded powerful models, it has left us without a unified framework to unde

Test-time regression: a unifying framework for designing sequence models with associative memory | HTML5

related reading