Laundry list · Tanishq Kumar
Please steal my AI research ideas. This is a list of research questions and concrete experiments I would like to see done. If you are looking to break into AI research (e.g. as an undergraduate, or a software engineer in industry), these are low hanging fruit. Shoot me a line with experimental results if you want to collaborate. When should you start a pretraining run from scratch vs from a past checkpoint? Should you lean toward the former over the latter as available compute goes to infinity? The more general question here is how best to use a previous-generation base model 𝐵 𝑇 when starting a pretraining run for a new generation of models 𝐵 𝑇 + 1 ? Figure 5[d] in this paper shows starting from an ImageNet-pretrained checkpoint becomes less helpful as compute increases in the setting of latent diffusion models for video generation. At the beginning of training, should you start with a distillation-type objective like logit-matching against 𝐵 𝑇 but then anneal into a pure n