Fully Async Training: In-flight Weight Update / Multi-Turn Partial Rollout
SkyRL supports fully async training for any generator that implements the GeneratorInterface, including your arbitrary agent harness (both single-turn and multi-turn), to address the straggler issues in long-horizon tasks. We treat "fully async training" synonymous to the terms "in-flight weight update" and "multi-turn partial rollout", as discussed in works like AReal, PipelineRL, ScaleRL, etc. The core logic of fully async training lives in fully_async_trainer.py, a subclass of the synchronous trainer class in trainer.py. This fully_async_trainer.py works out-of-the-box with any generator that the user implements -- so users do not need to implement any additional logic for fully async training. Overview of Fully Async Training in SkyRL The fully async training is currently only supported for Generators that use /chat/completions. We will support .generate() and /completions soon. There are three types of RL training we are interested in here, well-depicted by AReal's Figure 1 and Fi