Algorithms — Ray 2.9.3

From Ray 2.6.0 onwards, RLlib is adopting a new stack for training and model customization, gradually replacing the ModelV2 API and some convoluted parts of Policy API with the RLModule API. Click here for details. Tip Check out the environments page to learn more about different environment types. Algorithm Frameworks Discrete Actions Continuous Actions Multi-Agent Model Support Multi-GPU APPO tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch BC tf + torch Yes +parametric Yes Yes +RNN torch CQL tf + torch No Yes No tf + torch DreamerV3 tf Yes Yes No +RNN (GRU-based by default) tf DQN, Rainbow tf + torch Yes +parametric No Yes tf + torch IMPALA tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch MARWIL tf + torch Yes +parametric Yes Yes +RNN torch PPO tf + torch Yes +parametric Yes Yes +RNN, +LSTM auto-wrapping, +Attention, +autoreg tf + torch SAC tf + torch Yes Yes Yes torch Multi-Agent only Methods

Algorithms # The following table is an overview of all available algorithms in RLlib. Note that all algorithms support multi-GPU training on a single (GPU) node in Ray (open-source) ( ) as well as multi-GPU training on multi-node (GPU) clusters when using the Anyscale platform ( ). Algorithm Single- and Multi-agent Multi-GPU (multi-node) Action Spaces On-Policy PPO (Proximal Policy Optimization) Off-Policy DQN/Rainbow (Deep Q Networks) SAC (Soft Actor Critic) High-throughput on- and off policy APPO (Asynchronous Proximal Policy Optimization) IMPALA (Importance Weighted Actor-Learner Architectu

saved by

• Michael Frew

related reading

• The 37 Implementation Details of Proximal Policy Optimization · The ICLR Blog Trackiclr-blog-track.github.io
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• State of RL for reasoning LLMs | A. Weersaweers.de
• Algorithms - Spinning Up documentationspinningup.openai.com
• [1707.06347] Proximal Policy Optimization Algorithmsarxiv.org
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com