The Bitter Lesson is coming for Tokenization | ⛰️ lucalp

Highlights the desire to replace tokenization with a general method that better leverages compute and data. We'll see tokenization's fragility and review the Byte Latent Transformer arch.

The Bitter Lesson is coming for Tokenization – ⛰️ lucalp The Bitter Lesson is coming for Tokenization 24 Jun, 2025 a world of LLMs without tokenization is desirable and increasingly possible Published on 24/06/2025 • ⏱️ 29 min read In this post, we highlight the desire to replace tokenization with a general method that better leverages compute and data. We'll see tokenization's role, its fragility and we'll build a case for removing it. After understanding the design space, we'll explore the potential impacts of a recent promising candidate (Byte Latent Transformer) and build strong intuitions

saved by

• Aryan Naik
• Sebastian J Zhao
• Gloria Ma
• Sarah

related reading

• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Byte Latent Transformer: Patches Scale Better Than Tokens | Research - AI at Metaai.meta.com
• MambaByte: Token-free Selective State Space Modelarxiv.org
• Transformer Inference Arithmetic | kipply's blogkipp.ly
• 2502.11089arxiv.org
• Overleaf Examplearxiv.org
• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• H-Nets - the Past | Goomba Labgoombalab.github.io
• The Annotated Transformernlp.seas.harvard.edu
• Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com