Quadrillion-parameter transformers and how to train them – Q
Extrapolated to 5T tokens, our law prescribes ~7,000× Chinchilla's parameter count at 1029 FLOPs, and the count keeps growing with compute. Bigger models on repeated data need much more weight decay. Our hyperparameter law stays near-optimal at 16× the fitted compute. With scaled hyperparameters and ensembling, loss keeps falling with compute on fixed data. Prior recipes get worse. The 1,000,000x sample efficiency gap between AI and humans is one of the central mysteries of the field, with no good explanation. One clue is model size: Chinchilla’s prescription for 30T tokens is a ~1.5T-parameter model, a tokens per parameter of 20:1 (Hoffmann et al., 2022). Humans, by contrast, have ~100T parameters (counting each synapse as a weight) and learn from just ~200M tokens, a tokens per parameter of 1:500,000. The tokens per parameter gap between AI and humans is thus 10,000,000x, and an important step forward for the sample efficiency gap may be to close it. On 30T tokens, we should be train