naklecha/llama3-from-scratch: llama3 implementation one matrix multiplication at a time

in this file, i implemented llama3 from scratch, one tensor and matrix multiplication at a time. also, im going to load tensors directly from the model file that meta provided for llama3, you need to download the weights before running this file. here is the offical link to download the weights: https://llama.meta.com/llama-downloads/ im not going to implement a bpe tokenizer (but andrej karpathy has a really clean implementation) link to his implementation: https://github.com/karpathy/minbpe normally, reading this depends on how the model classes are written and the variable names inside them. but since we are implementing llama3 from scratch we will read the file one tensor at a time. here we use tiktoken (i think an openai library) as the tokenizer IM SORRY but this is the only part of the codebase where i use an inbuilt neural network module anyway, so our [17x1] tokens are now [17x4096], i.e. 17 embeddings (one for each token) of length 4096 note: keep track of the shapes, it mak

llama3 implemented from scratch in this file, i implemented llama3 from scratch, one tensor and matrix multiplication at a time. also, im going to load tensors directly from the model file that meta provided for llama3, you need to download the weights before running this file. here is the offical link to download the weights: https://llama.meta.com/llama-downloads/ tokenizer im not going to implement a bpe tokenizer (but andrej karpathy has a really clean implementation) link to his implementation: https://github.com/karpathy/minbpe from pathlib import Path import tiktoken from tiktoken . loa

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Llama 2 · Hugging Facehuggingface.co
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• How LLMs Actually Work | 0xkato0xkato.xyz
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• Understanding and Coding Self-Attention, Multi-Head Attention, Causal-Attention, and Cross-Attention in LLMsmagazine.sebastianraschka.com
• ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
• API Reference — TensorRT LLMnvidia.github.io
• Inference characteristics of Llama · Cursorcursor.com
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• How LLM Inference Worksarpitbhayani.me