Learning to Learn with Generative Models of Neural Network Checkpoints

wpeebles.com · 803 words · saved by 1 readers

We explore a data-driven approach for learning to optimize neural networks. We construct a dataset of neural network checkpoints and train a generative model on the parameters. In particular, our model is a conditional diffusion transformer that, given an initial input parameter vector and a prompted loss, error, or return, predicts the distribution over parameter updates that achieve the desired metric. At test time, it can optimize neural networks with unseen parameters for downstream tasks in just one update. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning. Over the last decade, the deep learning community has generated a massive amount of neural network checkpoints. They contain a wealth of information: diverse parameter configurations and rich metrics such as test losses, classification errors and reinforcement learning returns that describe the quality of the checkpoint. We pre-train a generative model on millions o

Learning to Learn with Generative Models of Neural Network Checkpoints Learning to Learn with Generative Models of Neural Network Checkpoints William Peebles* Ilija Radosavovic* Tim Brooks Alexei Efros Jitendra Malik William Peebles* Ilija Radosavovic* Tim Brooks Alexei Efros Jitendra Malik University of California, Berkeley Paper Code We explore a data-driven approach for learning to optimize neural networks. We construct a dataset of neural network checkpoints and train a generative model on the parameters. In particular, our model is a conditional diffusion transformer that, given an initia

Learning to Learn with Generative Models of Neural Network Checkpoints

related reading