What Matters for Simulation-to-Online Reinforcement Learning on Real Robots
Abstract: We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100+ real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. Rather than proposing a new algorithm, we study how standard off-policy RL can be made reliable in the sim-to-online setting. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yields stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort. We typically imagine learning agents as those capable of learning continuously and adapting “on-the-fly” from information as it arrives [46, 13]. Reinforcement learning (RL) formalizes how such agents can