Iterative RandOpt
tldr; The RandOpt algorithm introduced in the Neural Thickets paper is a fully parallelizable, iteration-free method. In this blog, we introduce Iterative RandOpt, which loops through random guessing → ensembling the top-k models → distilling them into one model. We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% accuracy, outperforming PPO, GRPO, and ES under a matched compute budget, while still being deployed as a single model. Code: https://github.com/sunrainyg/RandOpt/tree/iterative-randopt RandOpt is a fully parallelizable, iteration-free method for training LLMs. The algorithm is as follows: One limitation of RandOpt for practical use is ensembling: voting over the top-k models requires k times more forward passes compared with GRPO/PPO/ES. Distill top-k models into single model fixes this (see Sec. 7 in the neural thickets paper). Since the distilled model is now a better base than the one you started with, you can search around it again. RandOpt becomes
July 2026 · Yulu Gan, Phillip Isola tldr; The RandOpt algorithm introduced in the Neural Thickets paper is a fully parallelizable, iteration-free method. In this blog, we introduce Iterative RandOpt, which loops through random guessing → ensembling the top-k models → distilling them into one model. We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% accuracy, outperforming PPO, GRPO, and ES under a matched compute budget, while still being deployed as a single model. Code: https://github.com/sunrainyg/RandOpt/tree/iterative-randopt What is RandOpt? RandOpt is a…
saved by
related reading
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- State of RL for reasoning LLMs | A. Weersaweers.de
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
- [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- [2604.13010] Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillationarxiv.org
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- [2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelsarxiv.org
- [2507.19457] GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningarxiv.org