I built ChatGPT with Minecraft redstone! - YouTube
youtube.com · 249 words · saved by 3 readers
Follow along using the transcript. Follow along using the transcript.
I built a small language model in Minecraft using no command blocks or datapacks! The model has 5,087,280 parameters, trained in Python on the TinyChat dataset of basic English conversations. It has an embedding dimension of 240, vocabulary of 1920 tokens, and consists of 6 layers. The context window size is 64 tokens, which is enough for (very) short conversations. Most weights were quantized to 8 bits, although the embedding and LayerNorm weights are stored at 18 and 24 bits respectively. The quantized weights are linked below; they are split into hundreds of files corresponding to the…
saved by
related reading
- Let's build GPT: from scratch, in code, spelled out.youtube.com
- microgptkarpathy.github.io
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- GitHub - karpathy/nanochat: The best ChatGPT that $100 can buy.github.com
- Let's reproduce GPT-2 (124M)youtube.com
- [2306.00937] STEVE-1: A Generative Model for Text-to-Behavior in Minecraftarxiv.org
- Deep Dive into LLMs like ChatGPTyoutube.com
- A General Goal-Conditioned Minecraft Model - Pantographpantograph.com
- GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
- RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
- GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
- Quantization from the ground upngrok.com