The 4-bitter Lesson | humans&
We have developed and shared a low-precision RL recipe preserving higher-precision training dynamics. In this recipe, we needed to address instability from the forward pass due to policy quantization errors, from the backward pass due to gradient mismatches, and at the intersection of both due to a small set of particularly sensitive weights. We explain how we addressed each and validated the final recipe.
Introduction RL Training Simulator. The simulator models an asynchronous RL system where samplers continuously generate rollouts while the trainer updates the policy. Policy mismatch arises from both off-policyness (stale rollouts) and quantization error , accumulating into policy drift that eventually degrades reward if it exceeds the optimizer's correction capacity. RL & Efficiency Knobs. Off-policy, weight sync, batch size, and horizon control the degree of asynchrony and policy staleness. MXFP8 and NVFP4 improve training and rollout efficiency, while dequantized backward, BF16 last 15%, an
Explore this link on the map →related reading
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- Defeating Nondeterminism in LLM Inference - Thinking Machines Labthinkingmachines.ai
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studyarxiv.org
- Composer2.pdfcursor.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- Orbit - Ultra-efficient RL Pipelinespherelab.ai
- A Guide to Quantization in LLMs | Symbl.aisymbl.ai
- Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
- A Visual Guide to Quantization - by Maarten Grootendorstnewsletter.maartengrootendorst.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- RL Scaling Laws for LLMs - by Cameron R. Wolfe, Ph.D.cameronrwolfe.substack.com