flâneur

PyTorch Profiling 101 with Modded-NanoGPT

blog.underfit.ai · 2,355 words · saved by 2 readers

Understanding where training time is spent is essential for optimization. We can use the PyTorch profiler to capture traces that visualize the timeline of CPU operations, GPU kernel execution, and the coordination between them. In this post we'll walk through setting up the PyTorch profiler, using Modded-NanoGPT as an example, and learn how to navigate and interpret profiler traces. The Modded-NanoGPT project was started by @KellerJordan and implements a speedrun of the NanoGPT project by Andrej Karpathy. The goal of Modded-NanoGPT is to train the NanoGPT language model to get to below 3.28 cross-entropy loss on the FineWeb validation set as fast as possible. The speedrun is timed on 8 Nvidia H100 GPUs and there are two tracks, small (124M parameters) and medium (350M parameters), and both tracks started from the llm.c baseline. The small track started from a baseline of 45 minutes and through the combined effort of many it's now down to below 2.5 minutes! H100s are available on variou

PyTorch Profiling 101 with Modded-NanoGPT Understanding where training time is spent is essential for optimization. We can use the PyTorch profiler to capture traces that visualize the timeline of CPU operations, GPU kernel execution, and the coordination between them. In this post we'll walk through setting up the PyTorch profiler, using Modded-NanoGPT as an example, and learn how to navigate and interpret profiler traces. The Modded-NanoGPT project was started by @KellerJordan and implements a speedrun of the NanoGPT project by Andrej Karpathy . The goal of Modded-NanoGPT is to train the Nan

saved by

related reading