Efficiently Estimating Pareto Frontiers with Cyclic Learning Rate Schedules

Benchmarking the tradeoff between model accuracy and training time is computationally expensive. Cyclic learning rate schedules can construct a tradeoff curve in a single training run. These cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency.

Efficiently Estimating Pareto Frontiers with Cyclic Learning Rate Schedules | Databricks Blog Skip to main content Benchmarking the tradeoff between model accuracy and training time is computationally expensive. Cyclic learning rate schedules can construct a tradeoff curve in a single training run. These cyclic tradeoff curves can be used to evaluate the effects of algorithmic choices on network training efficiency. This work is has been posted as an arXiv preprint: Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Rates . Problem Statement: Producing Tradeoff Curves Efficie

saved by

• Saeejith Nair

related reading

• Mosaic ResNet Deep Dive | Databricks Blogmosaicml.com
• Making Deep Learning go Brrrr From First Principleshorace.io
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• [2310.07831] Optimal Linear Decay Learning Rate Schedules and Further Refinementsarxiv.org
• [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.github.com
• IsoCompute Playbook: Optimally Scaling Sampling Compute for RL Training of LLMscompute-optimal-rl-llm-scaling.github.io
• [1709.05011] ImageNet Training in Minutesarxiv-vanity.com
• The Continual Learning Problemjessylin.com
• The 4-bitter Lesson | humans&humansand.ai
• [2509.14786] Pre-training under infinite computearxiv.org
• Batch size 1 or ∞kothasuhas.github.io