[2605.23872] Training-Free Looped Transformers

Abstract:We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm transformer block as a forward Euler step on an ODE, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our method improves Qwen3-4B-Instruct by +2.64 pp on MMLU-Pro, Qwen3-30B-A3B-Instruct by +1.14 pp on CommonsenseQA, and Moonlight-16B-A3B-Instruct by +1.20 pp on OpenBookQA.

View PDF HTML (experimental) Abstract:We introduce training-free looped transformers, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by viewing a pre-norm…

saved by

• Anonymous Cow

related reading

• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inferencearxiv.org
• [2607.16051] Loop the Loopies!arxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Looped Transformers as Programmable Computersarxiv.org
• [2607.13491] DeepLoop: Depth Scaling for Looped Transformersarxiv.org
• [2604.11791] A Mechanistic Analysis of Looped Reasoning Language Modelsarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• The Annotated Transformernlp.seas.harvard.edu
• Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• LLMs Don't Need More Parameters. They Need Loops.youtube.com
• The Continual Learning Problemjessylin.com