[2601.08393] Controlled LLM Training on Spectral Sphere

Abstract:Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ($\boldsymbol{\mu}$P) provides a theoretical safeguard for width-invariant $\Theta(1)$ activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the \textbf{Spectral Sphere Optimizer (SSO)}, which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully $\boldsymbol{\mu}$P-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.

Controlled LLM Training on Spectral Sphere * Tian Xie1 Haoming Luo2 Haoyu Tang2 Yiwen Hu2 Jason Klein Liu4 Qingnan Ren1 Yang Wang1 Wayne Xin Zhao2,4 Rui Yan3 Bing Su2 Chong Luo1…

saved by

• Aidan Smith

related reading

• Modular Manifolds - Thinking Machines Labthinkingmachines.ai
• Deriving Muonjeremybernste.in
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
• The Practitioner’s Guide to the Maximal Update Parameterization - Cerebrascerebras.ai
• [2604.04891] Muon Dynamics as a Spectral Wasserstein Flowarxiv.org
• The Practitioner's Guide to the Maximal Update Parameterization | EleutherAI Blogblog.eleuther.ai
• A Spectral Condition for Feature Learningarxiv.org
• [2411.19870] DeMo: Decoupled Momentum Optimizationarxiv.org
• [2509.14786] Pre-training under infinite computearxiv.org
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org