✳flâneur — a map of the web's best reading

Training a compute-optimal gpt2-small – Tomek Korbak — personal homepage

tomekkorbak.com · 976 words · saved by 1 readers

Assume you’d like to train a gpt2-small-sized model (117m parameters). What is the optimal training set size? I’ll try to estimate that number following Training Compute-Optimal Large Language Models (also known as “the Chinchilla paper”). The Chinchilla paper presents a scaling law for language modelling loss L 𝐿 as a function of model size N 𝑁 (the number of parameters) and training dataset size D 𝐷 (the number of tokens). According to their model, language model’s loss is a sum of thee terms: Each term has an intuitive meaning: E 𝐸 is a constant roughly equal the entropy of natural language (or, whatever your training distribution is). An infinitely big model trained on infinitely many tokens would approach E 𝐸 . The first and second terms are penalties paid for, respectively, having a finite model and a finite dataset. (A discussion can be found here.) The Chinchilla paper paper estimates A=406.4 𝐴 = 406.4 , B=410.7 𝐵 = 410.7 , E=1.69 𝐸 = 1.69 , α=0.32 𝛼 = 0.3

Training a compute-optimal gpt2-small Assume you’d like to train a gpt2-small -sized model (117m parameters). What is the optimal training set size? I’ll try to estimate that number following Training Compute-Optimal Large Language Models (also known as “the Chinchilla paper”). Background: Chinchilla scaling law The Chinchilla paper presents a scaling law for language modelling loss \(L\) as a function of model size \(N\) (the number of parameters) and training dataset size \(D\) (the number of tokens). According to their model, language model’s loss is a sum of thee terms: \[L(N,D) = \frac{A}

Explore this link on the map →

related reading