Byte pair encoding

Byte pair encoding (also known as digram coding) is an algorithm, first described in 1994 by Philip Gage, for encoding strings of text into tabular form for use in downstream modeling. A slightly-modified version of the algorithm is used in large language model tokenizers.

Byte-pair encoding - Wikipedia Jump to content From Wikipedia, the free encyclopedia (Redirected from Byte pair encoding ) Adjacent characters (tokens) merge-based compression algorithm In computing , byte-pair encoding ( BPE ), [ 1 ] [ 2 ] or digram coding , [ 3 ] is an algorithm , first described in 1994 by Philip Gage, for encoding strings of text into smaller strings by creating and using a translation table. [ 4 ] A slightly modified version of the algorithm is used in large language model tokenizers. The original version of the algorithm focused on compres

saved by

• Micah Carroll

related reading

• Byte-Pair Encoding tokenization · Hugging Facehuggingface.co
• GitHub - karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. · GitHubgithub.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Can gzip be a language model?nathan.rs
• Compression and Intelligencegreene.sh
• [2309.10668] Language Modeling Is Compressionarxiv.org
• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• Sampling from Your Language Model One Byte at a Timearxiv.org
• 2025.acl-long.453.pdfaclanthology.org
• [2112.10508] Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLParxiv.org
• Data Compression Explainedmattmahoney.net
• [2605.22821] Tokenisation via Convex Relaxationsarxiv.org