[2210.14215] In-context Reinforcement Learning with Algorithm Distillation

We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data.

Preprint I N - CONTEXT R EINFORCEMENT L EARNING WITH A LGORITHM D ISTILLATION Michael Laskin∗ Luyu Wang∗ Junhyuk Oh Emilio Parisotto Stephen Spencer Richie Steigerwald DJ Strouse Steven Hansen Angelos Filos Ethan Brooks Maxime Gazeau Himanshu Sahni Satinder Singh Volodymyr…

related reading

• On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
• 2106.01345arxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• [2103.06326] S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• Exploration for the Efficient Deployment of Reinforcement Learning Agentsopenreview.net
• [1709.06560] Deep Reinforcement Learning that Mattersarxiv.org
• Self-Distillation Enables Continual Learningarxiv.org
• Decoupling Representation Learning from Reinforcement Learningarxiv.org
• [2601.18734] Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelsarxiv.org
• Contrastive Learning as Goal-Conditioned Reinforcement Learningarxiv.org