The 4-bitter Lesson | humans&
We have developed and shared a low-precision RL recipe preserving higher-precision training dynamics. In this recipe, we needed to address instability from the forward pass due to policy quantization errors, from the backward pass due to gradient mismatches, and at the intersection of both due to a small set of particularly sensitive weights. We explain how we addressed each and validated the final recipe.
Introduction RL Training Simulator. The simulator models an asynchronous RL system where samplers continuously generate rollouts while the trainer updates the policy. Policy mismatch arises from both off-policyness (stale rollouts) and quantization error , accumulating into policy drift that eventually degrades reward if it exceeds the optimizer's correction capacity. RL & Efficiency Knobs. Off-policy, weight sync, batch size, and horizon control the degree of asynchrony and policy staleness. MXFP8 and NVFP4 improve training and rollout efficiency, while dequantized backward, BF16 last 15%, an
saved by
related reading
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- Defeating the Training-Inference Mismatch via FP16arxiv.org
- Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
- Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
- How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- Debugging Reinforcement Learning Systemsandyljones.com
- [2510.11696] QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMsarxiv.org
- Speeding up RL with high-leverage samples | Applied Computeappliedcompute.com
- [2602.17616] Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMsarxiv.org
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev