google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models.

A playbook for systematically maximizing the performance of deep learning models.

This is not an officially supported Google product. Varun Godbole†, George E. Dahl†, Justin Gilmer†, Christopher J. Shallue‡, Zachary Nado† † Google Research, Brain Team ‡ Harvard University Table of Contents Who is this document for? Why a tuning playbook? Guide for starting a new project Choosing the model architecture Choosing the optimizer Choosing the batch size Choosing the initial configuration A scientific approach to improving model performance The incremental tuning strategy Exploration vs exploitation Choosing the goal for the next round of experiments Designing…

saved by

• Mehul Raheja
• Jacob G-W
• Ishan Mukherjee

related reading

• GitHub - google-research/tuning_playbook: A playbook for systematically maximizing the performance of deep learning models. · GitHubgithub.com
• The Practitioner’s Guide to the Maximal Update Parameterization - Cerebrascerebras.ai
• Hyperparameter optimization - Wikipediaen.wikipedia.org
• How To Scale Your Modeljax-ml.github.io
• Scaling Laws That Extrapolate 300× Past the Fitopenathena.ai
• Scaling Laws, Carefully | Lil'Loglilianweng.github.io
• The Little Book of Deep Learningfleuret.org
• Scaling Laws For Every Hyperparameter Via Cost-Aware HPO - Imbueimbue.com
• frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
• Evaluation and Fine Tuning | Virgiliovirgili0.github.io
• David Duvenaudcs.toronto.edu
• Understanding Deep Learningudlbook.github.io