Let's reproduce GPT-2 (124M) - YouTube

Follow along using the transcript. Follow along using the transcript. Sponsored New New

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90%…

saved by

• Freeman Jiang
• Sarah Pan
• Jackson Mowatt Gok

related reading

• Let's build GPT: from scratch, in code, spelled out.youtube.com
• GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
• Deep Dive into LLMs like ChatGPTyoutube.com
• microgptkarpathy.github.io
• Neural Networks: Zero To Herokarpathy.ai
• interpreting GPT: the logit lens — LessWronglesswrong.com
• GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.github.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
• GitHub - karpathy/autoresearch: AI agents running research on single-GPU nanochat training automaticallygithub.com
• GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs. · GitHubgithub.com
• The Annotated Transformernlp.seas.harvard.edu