flâneur

IsoFLOP curves of large language models are extremely flat – Severely Theoretical

severelytheoretical.wordpress.com · 1,040 words · saved by 1 readers

An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchil…

An interesting detail in the recently released Llama-3 technical report has caught my eye (p. 8): This has caught my eye, since I had noted the same phenomenon in a previous post about the Chinchilla scaling laws (more than two years ago) to argue for training smaller models (point 4 in that post). I’m glad that this observation is finally being taken seriously, but I think the quotation above from the Llama-3 paper still underestimates the extent of this isoFLOP flatness issue. The performance of these models is not just robust to small variations in model size around the optimal, but it…

saved by

related reading