Mosaic ResNet Deep Dive
TL;DR: We recently released a set of recipes which can accelerate training of a ResNet-50 on ImageNet by up to 7x over standard baselines. In this report we take a deep dive into the technical details of our work and share the insights we gained about optimizing the efficiency of model training over a broad range of compute budgets.
Mosaic ResNet Deep Dive | Databricks Blog Skip to main content TL;DR: We recently released a set of recipes which can accelerate training of a ResNet-50 on ImageNet by up to 7x over standard baselines. In this report we take a deep dive into the technical details of our work and share the insights we gained about optimizing the efficiency of model training over a broad range of compute budgets. Introduction ResNets have established themselves as the go-to baseline and testbed for computer vision research (see ResNet Strikes Back or the PyTorch blog ). More efficient training recipes can save m
saved by
related reading
- Efficiently Estimating Pareto Frontiers with Cyclic Learning Rate Schedules | Databricks Blogmosaicml.com
- ♻️ Auto Microbatching - Composerdocs.mosaicml.com
- A Recipe for Training Neural Networkskarpathy.github.io
- A Recipe for Training Neural Networkskarpathy.github.io
- Making Deep Learning go Brrrr From First Principleshorace.io
- [1709.05011] ImageNet Training in Minutesarxiv-vanity.com
- Training LLMs with AMD MI250 GPUs and MosaicML | Databricks Blogmosaicml.com
- Mosaic LLMs: GPT-3 quality formosaicml.com
- 1.5x faster MoE training with custom MXFP8 kernels · Cursorcursor.com
- The Best GPUs for Deep Learning in 2023 — An In-depth Analysistimdettmers.com
- >10x More Efficient Pretraining — Magicmagic.dev
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com