Scaling law tutorial
Scaling laws are the science behind 1., are what justified the huge investments in compute and are key to the pretraining of current models. They are interesting from an engineering perspective (useful tools to design models), and from a scientific standpoint (suggest some universal principles underlying learning to be discovered!). Build intuition on why scaling laws are useful and why they hold. In modern large language models, there are many training stages; we focus on pretraining. Context in, next-token distribution out. At inference: sample, append, repeat. (Banko & Brill, 2001)(Hestness et al., 2017)(Rosenfeld et al., 2020) (Kaplan et al., 2020) (Hoffmann et al., 2022) We review some of the results from (Kaplan et al., 2020) and (Hoffmann et al., 2022) in this part and provide some pointers to extensions. The variables we care about are: NNN, DDD and CCC are actually coupled: we will now show that C=6NDC = 6NDC=6ND. A linear layer does y=Wxy = Wxy=Wx, where W∈Rm×nW\in\mathbb{R}^