Algorithms — Ray 2.9.3

From Ray 2.6.0 onwards, RLlib is adopting a new stack for training and model customization, gradually replacing the ModelV2 API and some convoluted parts of Policy API with the RLModule API. Click here for details. Tip Check out the environments page to learn more about different environment types. Algorithm Frameworks Discrete Actions Continuous Actions Multi-Agent Model Support Multi-GPU APPO tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch BC tf + torch Yes +parametric Yes Yes +RNN torch CQL tf + torch No Yes No tf + torch DreamerV3 tf Yes Yes No +RNN (GRU-based by default) tf DQN, Rainbow tf + torch Yes +parametric No Yes tf + torch IMPALA tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch MARWIL tf + torch Yes +parametric Yes Yes +RNN torch PPO tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch SAC tf + torch Yes Yes Yes torch Multi-Agent only Methods

Algorithms # The following table is an overview of all available algorithms in RLlib. Note that all algorithms support multi-GPU training on a single (GPU) node in Ray (open-source) ( ) as well as multi-GPU training on multi-node (GPU) clusters when using the Anyscale platform ( ). Algorithm Single- and Multi-agent Multi-GPU (multi-node) Action Spaces On-Policy PPO (Proximal Policy Optimization) Off-Policy DQN/Rainbow (Deep Q Networks) SAC (Soft Actor Critic) High-throughput on- and off policy APPO (Asynchronous Proximal Policy Optimization) IMPALA (Importance Weighted Actor-Learner Architectu

saved by

• Michael Frew

related reading

• 2109.10813.pdfarxiv.org
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• Keep the Tokens Flowing: Lessons from 16 Open-Source RL Librarieshuggingface.co
• State of RL for reasoning LLMs | A. Weersaweers.de
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
• Algorithms - Spinning Up documentationspinningup.openai.com
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• [2606.05555] Representation Learning Enables Scalable Multitask Deep Reinforcement Learningarxiv.org