Byte pair encoding
Byte pair encoding (also known as digram coding) is an algorithm, first described in 1994 by Philip Gage, for encoding strings of text into tabular form for use in downstream modeling. A slightly-modified version of the algorithm is used in large language model tokenizers.
Byte-pair encoding - Wikipedia Jump to content From Wikipedia, the free encyclopedia (Redirected from Byte pair encoding ) Adjacent characters (tokens) merge-based compression algorithm In computing , byte-pair encoding ( BPE ), [ 1 ] [ 2 ] or digram coding , [ 3 ] is an algorithm , first described in 1994 by Philip Gage, for encoding strings of text into smaller strings by creating and using a translation table. [ 4 ] A slightly modified version of the algorithm is used in large language model tokenizers. The original version of the algorithm focused on compres
saved by
related reading
- Byte-Pair Encoding tokenization · Hugging Facehuggingface.co
- The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
- GitHub - karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. · GitHubgithub.com
- Compression and Intelligencegreene.sh
- Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
- Can gzip be a language model?nathan.rs
- [2309.10668] Language Modeling Is Compressionarxiv.org
- The End of Tokenizationronaldyu.substack.com
- Sampling from Your Language Model One Byte at a Timearxiv.org
- LLMs can invent their own compression - Rajan Agarwalrajan.sh
- [2605.22821] Tokenisation via Convex Relaxationsarxiv.org
- 2025.acl-long.453.pdfaclanthology.org