GPT in 60 Lines of NumPy | Jay Mody
jaykmody.com · 9,202 words · saved by 5 readers
Implementing a GPT model from scratch in NumPy.
GPT in 60 Lines of NumPy | Jay Mody GPT in 60 Lines of NumPy January 30, 2023 In this post, we'll implement a GPT from scratch in just 60 lines of numpy . We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text. Note: This post assumes familiarity with Python, NumPy, and some basic experience with neural networks. This implementation is for educational purposes, so it's missing lots of features/improvements on purpose to keep it as simple as possible while remaining complete. All the code for this blog post can be found at github.com/ja
saved by
related reading
- Let's build GPT: from scratch, in code, spelled out.youtube.com
- An Intuition for Attention | Jay Modyjaykmody.com
- Let's reproduce GPT-2 (124M)youtube.com
- microgptkarpathy.github.io
- Deep Dive into LLMs like ChatGPTyoutube.com
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- gpt-4.pdfcdn.openai.com
- interpreting GPT: the logit lens — LessWronglesswrong.com
- The Annotated Transformernlp.seas.harvard.edu
- Neural Networks: Zero To Herokarpathy.ai
- microgptgist.github.com
- Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io