RL without TD learning – The Berkeley Artificial Intelligence Research Blog

The BAIR Blog

In this post, I’ll introduce a reinforcement learning (RL) algorithm based on an “alternative” paradigm: divide and conquer . Unlike traditional methods, this algorithm is not based on temporal difference (TD) learning (which has scalability challenges ), and scales well to long-horizon tasks. We can do Reinforcement Learning (RL) based on divide and conquer, instead of temporal difference (TD) learning. Problem setting: off-policy RL Our problem setting is off-policy RL . Let’s briefly review what this means. There are two classes of algorithms in RL: on-policy RL and off-policy RL. On-policy

related reading

• RL without TD learningseohong.me
• Seohong Park on X: "We scaled up an "alternative" paradigm in RL: *divide and conquer*. Compared to Q-learning (TD learning), divide and conquer can naturally scale to much longer horizons. Blog post: https://t.co/xtXBzya0bI Paper: https://t.co/nqYkLucsWu ↓ https://t.co/XCdgUzaLxF" / Xx.com
• Q-learning is not yet scalableseohong.me
• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Scaling Laws for Value-Based RLvalue-scaling.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• The Promise of Hierarchical Reinforcement Learningthegradient.pub
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• RUDDER - Reinforcement Learning with Delayed Rewards | rudderml-jku.github.io
• Part 2: Kinds of RL Algorithms - Spinning Up documentationspinningup.openai.com
• State of RL for reasoning LLMs | A. Weersaweers.de