[2607.16051] Loop the Loopies!

Abstract:We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

Loop the Loopies! Zitian Gao Yilong Chen Yihao Xiao Xinyu Yang Ran Tao Joey Zhou Bryan Dai IQuest Research See the full author contributions here. Models: Loopie-20B-A2B…

saved by

• Arya G

related reading

• [2509.14786] Pre-training under infinite computearxiv.org
• Training-Free Looped Transformersarxiv.org
• [2607.13491] DeepLoop: Depth Scaling for Looped Transformersarxiv.org
• [2604.11791] A Mechanistic Analysis of Looped Reasoning Language Modelsarxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Looped Transformers as Programmable Computersarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• [2512.23675] End-to-End Test-Time Training for Long Contextarxiv.org
• LLMs Don't Need More Parameters. They Need Loops.youtube.com
• [2602.05910] Chunky Post-Training: Data Driven Failures of Generalizationarxiv.org
• GitHub - huskydoge/Awesome-Loop-Models: A curated list of papers and selected technical blogs on Loop Models.github.com
• [2605.15220] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Timearxiv.org