Training NanoGPT in 39.9 Seconds — Hyperstition

hyperstition.cc · 1,651 words · saved by 4 readers

How I brought NanoGPT training from 73.889 to 39.914 seconds with ANVIL II, sampled softmax, and sparse updates to the bigram and trigram tables.

Authors Deven Pietrzak. TL;DR We train the NanoGPT model to the target loss in 39.914 seconds on 8×H100, down from the previous record of 73.889 seconds. By percentage reduction in training time, this is a greater improvement than the previous 45 world records combined. We derive gains from ANVIL II, sampled softmax and a larger sparse n-gram model. For reasons proprietary to Hyperstition, we redacted methods that scale to the frontier from our public record, apart from ANVIL II. Learn more about its extensions: Cutting Pretraining Costs by 62%. Introduction The NanoGPT speedrun…

Training NanoGPT in 39.9 Seconds — Hyperstition

saved by

related reading