Let's build GPT: from scratch, in code, spelled out. - YouTube
youtube.com · 11,554 words · saved by 6 readers
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections t...
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video. Links: - Google colab for the video:…
saved by
related reading
- GPT in 60 Lines of NumPy | Jay Modyjaykmody.com
- Let's reproduce GPT-2 (124M)youtube.com
- GitHub - rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in PyTorch from scratch, step by stepgithub.com
- microgptkarpathy.github.io
- Neural Networks: Zero To Herokarpathy.ai
- Deep Dive into LLMs like ChatGPTyoutube.com
- A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
- What Is ChatGPT Doing … and Why Does It Work?-Stephen Wolfram Writingswritings.stephenwolfram.com
- The Annotated Transformernlp.seas.harvard.edu
- Transformers Laid Outgoyalpramod.github.io
- GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.github.com
- microgptgist.github.com