flâneur

How We Build Trillion Parameter Reasoning RL with 10% GPUs

macaron.im · 2,380 words · saved by 1 readers

We present what we believe is the first end-to-end Reinforcement Learning (RL) with Low-Rank Adaptor (LoRA) on a trillion-parameter reasoning model. Our system runs on large Mixture-of-Experts (MoE) models with 10% GPUs compared to conventional full-parameter RL. Our solutions have also been contributed to major open-source projects: NVIDIA’s Megatron-Bridge and Volcengine’s verl. This post has two main claims: Practical LoRA RL at Trillion-Parameter Scale. We detail the failure modes inherent to LoRA training on trillion-parameter MoE architectures and demonstrate how our hybrid-parallel engine, integrating verl and Megatron-Bridge, resolves these issues to enable stable and efficient RL. Why RL on Trillion-Parameter Models Pays Off. We prove that under similar compute budgets, LoRA RL on a trillion-parameter model outperforms full-parameter RL on much smaller models. In other words, when you already have a strong prior, spending RL compute on the largest model could be more effective

How We Build Trillion Parameter Reasoning RL with 10% GPUs Overview We present what we believe is the first end-to-end Reinforcement Learning (RL) with Low-Rank Adaptor (LoRA) on a trillion-parameter reasoning model . Our system runs on large Mixture-of-Experts (MoE) models with 10% GPUs compared to conventional full-parameter RL. Our solutions have also been contributed to major open-source projects: NVIDIA’s Megatron-Bridge and Volcengine’s verl . This post has two main claims: Practical LoRA RL at Trillion-Parameter Scale. We detail the failure modes inherent to LoRA training on trillion-pa

saved by

related reading