flâneur

Why test-time training? – Rabbitholes

sarahpannn.github.io · 1,468 words · saved by 1 readers

I was first introduced to test-time training (TTT) by Ekin Akyürek et al.’s SoTA performance on ARC-AGI (Akyürek et al. 2024). This idea was nonintuitive but compelling nonetheless, so I structured my deep learning final project around adapting pre-trained language models with Titan modules for long-context memory efficiency. Within an hour of training on a single GPU, updates made at test time had already recovered much of the performance gap between full-fidelity attention and a less expressive version. Almost a year later, I have gone through multiple iterations of this project, trying out new architecture changes and training methods. Still, I find myself asking fundamental questions about this research direction. Why should gradients be computed at test time? What limits on learning should the phrasing of test time imply? Why was I working on extending context lengths when agent harnesses had \compact as a way to summarize the entire token window? Why spend time on TTT when KV cac

I was first introduced to test-time training (TTT) by Ekin Akyürek et al.’s SoTA performance on ARC-AGI (Akyürek et al. 2024). This idea was nonintuitive but compelling nonetheless, so I structured my deep learning final project around adapting pre-trained language models with Titan modules for long-context memory efficiency. Within an hour of training on a single GPU, updates made at test time had already recovered much of the performance gap between full-fidelity attention and a less expressive version. Almost a year later, I have gone through multiple iterations of this project, trying…

saved by

related reading