Test-Time Training Done Right

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. Test-Time Training (TTT) models context dependencies by adapting part of the model’s weights (often referred to as fast weights) at inference time. This adapted fast weight, similar to recurrent states in RNNs, stores temporary memories of past tokens in the current sequence. Existing TTT methods have struggled to demonstrate effectiveness in handling long-sequence data, due to their computational inefficiency on modern GPUs. The TTT layers in many of these approaches operate with extremely low FLOPs utilization (often below 5%) because they deliberately apply small online mini-batch sizes (e.g., updating fast weights every 16 or 64 tokens). Moreover, a small mini-batch implies fine-grained block-wise causal dependencies in the data, making them unsuitab

Test-Time Training Done Right Tianyuan Zhang 1 Sai Bi 2 Yicong Hong 2 Kai Zhang 2 Fujun Luan 2 Songlin Yang 1 Kalyan Sunkavalli 2 William T. Freeman 1 Hao Tan 2 1 Massachusetts Institute of Technology 2 Adobe Research Abstract Test-Time Training (TTT) models context dependencies by adapting part of the model’s weights (often referred to as fast weights) at inference time. This adapted fast weight, similar to recurrent states in RNNs, stores temporary memories of past tokens in the current sequence. Existing TTT methods have struggled to demonstrate effectiveness in handling long-sequence data,

related reading

• Reimagining LLM Memory: Using Context as Training Data Unlocks Models That Learn at Test-Time | NVIDIA Technical Blogdeveloper.nvidia.com
• [2407.04620] Learning to (Learn at Test Time): RNNs with Expressive Hidden Statesar5iv.labs.arxiv.org
• Learning to (Learn at Test Time): RNNs with Expressive Hidden Statesarxiv.org
• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• [2501.00663] Titans: Learning to Memorize at Test Timearxiv.org
• microgptkarpathy.github.io
• The Annotated Transformernlp.seas.harvard.edu
• The Little Book of Deep Learningfleuret.org
• Extending Context is Hard | kaiokendevkaiokendev.github.io
• Linear Attention Fundamentals | Hailey Schoelkopfhaileyschoelkopf.github.io