Center for Responsible, Decentralized Intelligence at Berkeley

We introduce DELTA: a controlled suite of synthetic programming families with fully OOD splits and verifiable rewards. DELTA lets us ask two crisp questions: Learnability (can RL solve families where the base model has pass@K=0?) and Transferability (do the learned procedures generalize?) On several pass@128=0 families, RL exhibits a grokking-like phase transition: after a long near-zero-reward plateau, accuracy snaps to ~100%. That is discovery, not mere sharpening. A two-phase reward schedule is key: dense per-test rewards to escape the “all-zero” region, then binary full-pass to consolidate exact solutions. Binary-only gets stuck; dense-only hovers at “almost right.” The schedule yields the grokking jump. Transfer is selective: RL-trained policies recompose programming sub-skills and extrapolate to harder parametric regimes, but struggle on transformative shifts that require new invariants. Manufactoria is an old Flash game from 2010 where you test robots by reading colored tapes. W

RL Grokking Recipe: How Can We Enable LLMs to Solve Previously Unsolvable Tasks with RL? Yiyou Sun¹, Yuhan Cao, Pohao Huang¹, Haoyue Bai², Hannaneh Hajishirzi³⁴, Nouha Dziri⁴♠, Dawn Song¹♠ ¹ University of California, Berkeley · ² University of Wisconsin–Madison · ³ University of Washington · ⁴AI2 (♠ indicates equal advising) 💡 Question: Can reinforcement learning (RL) actually teach large language models new algorithms—or does it only "sharpen" what's already latent in the base model? Recent analyses say RL stays on a leash: pass@1 goes up, but what's possible at large sampling (e.g., pass@12

related reading

• DeepSeek-R1arxiv.org
• State of RL for reasoning LLMs | A. Weersaweers.de
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Composer2.pdfcursor.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com
• Explore | alphaXivalphaxiv.org
• How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWronglesswrong.com
• How to Explore to Scale RL Training of LLMs on Hard Problems? – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• Optimizing LLM Test-Time Compute Involves Solving a Meta-RL Problem – Machine Learning Blog | ML@CMU | Carnegie Mellon Universityblog.ml.cmu.edu
• [2506.19733] Breaking Barriers: Do Reinforcement Post Training Gains Transfer To Unseen Domains?arxiv.org