GPT in 60 Lines of NumPy | Jay Mody

Implementing a GPT model from scratch in NumPy.

GPT in 60 Lines of NumPy | Jay Mody GPT in 60 Lines of NumPy January 30, 2023 In this post, we'll implement a GPT from scratch in just 60 lines of numpy . We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text. Note: This post assumes familiarity with Python, NumPy, and some basic experience with neural networks. This implementation is for educational purposes, so it's missing lots of features/improvements on purpose to keep it as simple as possible while remaining complete. All the code for this blog post can be found at github.com/ja

saved by

• Ethan Benjamin
• Freeman Jiang
• Saeejith Nair
• Nima Pourjafar
• Anderson Hao

related reading

• microgptkarpathy.github.io
• interpreting GPT: the logit lens — LessWronglesswrong.com
• What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
• microgptgist.github.com
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• gpt-4.pdfcdn.openai.com
• GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
• The Annotated Transformernlp.seas.harvard.edu
• ali (@waterloo_intern) on Xx.com
• interpreting GPT: the logit lens — AI Alignment Forumalignmentforum.org
• GitHub - karpathy/nanoGPT: The simplest, fastest repository for training/finetuning medium-sized GPTs. · GitHubgithub.com
• Five years of GPT progressfinbarr.ca