microgpt
karpathy.github.io · 6,521 words · saved by 6 readers
Musings of a Computer Scientist.
This is a brief guide to my new art project microgpt , a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT. This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency. I cannot simplify this any further. This script is the culmination of multiple projects (micrograd, makemore, nanogpt, etc.) and a decade-long obsession to simplify LLMs to their bare essentials, an
saved by
related reading
- microgptgist.github.com
- GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- The Little Book of Deep Learningfleuret.org
- interpreting GPT: the logit lens — LessWronglesswrong.com
- frontier model training methodologies | Alex Wa's Blogdjdumpling.github.io
- ali (@waterloo_intern) on Xx.com
- GenAI Handbookgenai-handbook.github.io
- The Annotated Transformernlp.seas.harvard.edu
- Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
- How LLMs Actually Work | 0xkato0xkato.xyz