How We Build Trillion Parameter Reasoning RL with 10% GPUs

We present what we believe is the first end-to-end Reinforcement Learning (RL) with Low-Rank Adaptor (LoRA) on a trillion-parameter reasoning model. Our system runs on large Mixture-of-Experts (MoE) models with 10% GPUs compared to conventional full-parameter RL. Our solutions have also been contributed to major open-source projects: NVIDIA’s Megatron-Bridge and Volcengine’s verl. This post has two main claims: Practical LoRA RL at Trillion-Parameter Scale. We detail the failure modes inherent to LoRA training on trillion-parameter MoE architectures and demonstrate how our hybrid-parallel engine, integrating verl and Megatron-Bridge, resolves these issues to enable stable and efficient RL. Why RL on Trillion-Parameter Models Pays Off. We prove that under similar compute budgets, LoRA RL on a trillion-parameter model outperforms full-parameter RL on much smaller models. In other words, when you already have a strong prior, spending RL compute on the largest model could be more effective

How We Build Trillion Parameter Reasoning RL with 10% GPUs Overview We present what we believe is the first end-to-end Reinforcement Learning (RL) with Low-Rank Adaptor (LoRA) on a trillion-parameter reasoning model . Our system runs on large Mixture-of-Experts (MoE) models with 10% GPUs compared to conventional full-parameter RL. Our solutions have also been contributed to major open-source projects: NVIDIA’s Megatron-Bridge and Volcengine’s verl . This post has two main claims: Practical LoRA RL at Trillion-Parameter Scale. We detail the failure modes inherent to LoRA training on trillion-pa

saved by

• Dhruv Gautam

related reading

• LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
• [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
• Learning to Reason in 13 Parametersarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• RL at 1T Scale: prime-rl Performance Deep Diveprimeintellect.ai
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
• [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Orbit - Ultra-efficient RL Pipelinespherelab.ai
• [2506.06105] Text-to-LoRA: Instant Transformer Adaptionarxiv.org
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io