Iterative RandOpt
tldr; The RandOpt algorithm introduced in the Neural Thickets paper is a fully parallelizable, iteration-free method. In this blog, we introduce Iterative RandOpt, which loops through random guessing → ensembling the top-k models → distilling them into one model. We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% accuracy, outperforming PPO, GRPO, and ES under a matched compute budget, while still being deployed as a single model. Code: https://github.com/sunrainyg/RandOpt/tree/iterative-randopt RandOpt is a fully parallelizable, iteration-free method for training LLMs. The algorithm is as follows: One limitation of RandOpt for practical use is ensembling: voting over the top-k models requires k times more forward passes compared with GRPO/PPO/ES. Distill top-k models into single model fixes this (see Sec. 7 in the neural thickets paper). Since the distilled model is now a better base than the one you started with, you can search around it again. RandOpt becomes
July 2026 · Yulu Gan, Phillip Isola tldr; The RandOpt algorithm introduced in the Neural Thickets paper is a fully parallelizable, iteration-free method. In this blog, we introduce Iterative RandOpt, which loops through random guessing → ensembling the top-k models → distilling them into one model. We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% accuracy, outperforming PPO, GRPO, and ES under a matched compute budget, while still being deployed as a single model. Code: https://github.com/sunrainyg/RandOpt/tree/iterative-randopt What is RandOpt? RandOpt is a…
saved by
related reading
- LoRA Without Regret - Thinking Machines Labthinkingmachines.ai
- On-Policy Distillation - Thinking Machines Labthinkingmachines.ai
- Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
- GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
- State of RL for reasoning LLMs | A. Weersaweers.de
- Composer2.pdfcursor.com
- IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
- Tinkerthinkingmachines.ai
- LLM Resourcesforrestbicker.com
- [2602.19362] LLMs Can Learn to Reason Via Off-Policy RLarxiv.org
- LLM Optimization via Synthetic Distillationanarchyai.substack.com
- SFT, RL, and On-Policy Distillation Through a Distributional Lens | whnrehiew.github.io