Byte pair encoding

Byte pair encoding (also known as digram coding) is an algorithm, first described in 1994 by Philip Gage, for encoding strings of text into tabular form for use in downstream modeling. A slightly-modified version of the algorithm is used in large language model tokenizers.

Byte-pair encoding - Wikipedia Jump to content From Wikipedia, the free encyclopedia (Redirected from Byte pair encoding ) Adjacent characters (tokens) merge-based compression algorithm In computing , byte-pair encoding ( BPE ), [ 1 ] [ 2 ] or digram coding , [ 3 ] is an algorithm , first described in 1994 by Philip Gage, for encoding strings of text into smaller strings by creating and using a translation table. [ 4 ] A slightly modified version of the algorithm is used in large language model tokenizers. The original version of the algorithm focused on compres

saved by

• Micah Carroll

related reading

• GitHub - karpathy/minbpe: Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. · GitHubgithub.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Can gzip be a language model?nathan.rs
• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• [2309.10668] Language Modeling Is Compressionarxiv.org
• [2605.22821] Tokenisation via Convex Relaxationsarxiv.org
• Learning to reason with LLMs | OpenAIopenai.com
• MambaByte: Token-free Selective State Space Modelarxiv.org
• Variable-length encoding - Wikipediaen.wikipedia.org
• Data Compression Explainedmattmahoney.net
• H-Nets - the Past | Goomba Labgoombalab.github.io
• The Annotated Transformernlp.seas.harvard.edu