[2512.23675] End-to-End Test-Time Training for Long Context
Abstract:We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.
End-to-End Test-Time Training for Long Context Arnuv Tandon∗1,3 , Karan Dalal∗1,4 , Xinhao Li∗5 , Daniel Koceja∗3 , Marcel Rød∗3 , Sam Buchanan4 , Xiaolong Wang5 , Jure Leskovec3 , Sanmi Koyejo3 , Tatsunori Hashimoto3 , Carlos Guestrin3 , Jed McCaleb1 , Yejin Choi2 , Yu Sun∗2,3 1…
saved by
related reading
- Learning to (Learn at Test Time): RNNs with Expressive Hidden Statesarxiv.org
- Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
- Why test-time training? – Rabbitholessarahpannn.github.io
- Extending Context is Hard | kaiokendevkaiokendev.github.io
- [2507.04239] Scaling Context Requires Rethinking Attentionarxiv.org
- Test-Time Training Done Rightarxiv.org
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- A short note on some aspects of long context attention | nor's blognor-blog.pages.dev
- [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
- GLM-5.2: Built for Long-Horizon Tasksz.ai
- Why We Need Continual Learning | Andreessen Horowitza16z.com
- [2108.12409] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationarxiv.org