RL's Deadly Triad Meets Optimization | helen qu

The unsavory combination of function approximation, off-policy learning, and bootstrapping, or the so-called “deadly triad” of reinforcement learning, has a status approaching that of folklore for its ability to induce unstable learning dynamics/divergence in even the simplest of systems. While likely a household name among RL enthusiasts, I’ve seen surprisingly little discussion on its origins or first principles. I made this post to shed light on the deadly triad from the perspective of traditional optimization theory, with the goal of demonstrating that the underlying principles are nothing more exotic than well-established convergence guarantees for dynamical systems. If the deadly triad is known to cause instability, why not avoid it altogether? While today’s RL post-training pipelines rarely combine all three elements at once, each piece still survives for good reason: function approximation for scale, off-policy updates for data efficiency, and bootstrapping for fast learning. W

RL's Deadly Triad Meets Optimization | helen qu \( \renewcommand{\vec}[1]{\mathbf{#1}} \newcommand{\avec}{\vec{a}} \newcommand{\bvec}{\vec{b}} \newcommand{\cvec}{\vec{c}} \newcommand{\dvec}{\vec{d}} \newcommand{\evec}{\vec{e}} \newcommand{\fvec}{\vec{f}} \newcommand{\gvec}{\vec{g}} \newcommand{\hvec}{\vec{h}} \newcommand{\ivec}{\vec{i}} \newcommand{\jvec}{\vec{j}} \newcommand{\kvec}{\vec{k}} \newcommand{\lvec}{\vec{l}} \newcommand{\mvec}{\vec{m}} \newcommand{\nvec}{\vec{n}} \newcommand{\ovec}{\vec{o}} \newcommand{\pvec}{\vec{p}} \newcommand{\qvec}{\vec{q}} \newcommand{\rvec}{\vec{r}} \newcomma

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Debugging Reinforcement Learning Systemsandyljones.com
• Evolution as Backstop for Reinforcement Learning · Gwern.netgwern.net
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• RL without TD learningseohong.me
• Reward is not the optimization target — LessWronglesswrong.com
• Policy Gradient Algorithms | Lil'Loglilianweng.github.io
• Q-learning is not yet scalableseohong.me
• [1903.08894] Towards Characterizing Divergence in Deep Q-Learningarxiv.org
• RL without TD learning – The Berkeley Artificial Intelligence Research Blogbair.berkeley.edu