RL Pet Peeves Part 1 · Auriel
aurielws.github.io · 3,448 words · saved by 2 readers
My personal do's and don'ts for startups post-training their own model. A guide to trajectory eyeballing, harness debugging, and RL failure modes.
RL Pet Peeves Part 1 · Auriel This mini-series is a collection of rants on RL from my POV. Pure, unfiltered, first-person opinions from someone who's spent years deep in the trenches of pre-training, post-training/fine-tuning, inference time, and every layer of the stack for models from small distilled models (Pixel Real Tone base model) to frontier systems (Gemini + Nano Banana + Human Detection Models that powered Google Search, Waymo, Vertex AI). I've eyeballed thousands of trajectories, judged parametric wins and losses until my eyes bled at 2AM, and sat through more "data" pitches
saved by
related reading
- How to Post-Train · RL Fundamentals Mini-Serieshowtoposttrain.com
- Debugging Reinforcement Learning Systemsandyljones.com
- Tips for Empirical Alignment Research — AI Alignment Forumalignmentforum.org
- Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
- Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
- A Taxonomy of RL Environments for LLM Agentsleehanchung.github.io
- Simulated Users & Sad LLMs1a3orn.com
- Training Imperativesdan.io
- RL Environments and RL for Science: Data Foundries and Multi-Agent Architecturesnewsletter.semianalysis.com
- Low Latency and Model Training at Modalrhea24.github.io
- Is Frontier Asynchronous RL Solved? — Luke J. Huangluk-huang.github.io
- What if RL Environments Aren't Mispriced?benanderson.work