✳flâneur — a map of the web's best reading

karpathy/nanochat: The best ChatGPT that $100 can buy.

github.com · 2,250 words · saved by 3 readers

This repo is a full-stack implementation of an LLM like ChatGPT in a single, clean, minimal, hackable, dependency-lite codebase. nanochat is designed to run on a single 8XH100 node via scripts like speedrun.sh, that run the entire pipeline start to end. This includes tokenization, pretraining, finetuning, evaluation, inference, and web serving over a simple UI so that you can talk to your own LLM just like ChatGPT. nanochat will become the capstone project of the course LLM101n being developed by Eureka Labs. To get a sense of the endpoint of this repo, you can currently find nanochat d32 hosted on nanochat.karpathy.ai. "d32" means that this model has 32 layers in the Transformer neural network. This model has 1.9 billion parameters, it was trained on 38 billion tokens by simply running the single script run1000.sh, and the total cost of training was ~$800 (about 33 hours training time on 8XH100 GPU node). While today this is enough to outperform GPT-2 of 2019, it falls dramatically sh

Explore this link on the map →

saved by

related reading