The End of Tokenization - Ronald Yu's Substack

Some thoughts on tokenization and Meta's “Byte Latent Transformer: Patches Scale Better Than Tokens” paper

Meta’s new paper “Byte Latent Transformer: Patches Scale Better Than Tokens” eliminates LLMs’ dirtiest secret: tokenization. Instead of using a tokenizer, the Byte Latent Transformer, as the name implies, operates on the bytes of the raw text data. Previous works on eliminating tokenization have largely framed the central challenge as mitigating the fact that naively operating on raw bytes results in longer sequences and therefore extra compute for the same task. However, this is a losing battle as wasted compute will overshadow any minor benefits. The Byte Latent Transformer flips the…

saved by

• Gene Yang

related reading

• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• 2025.acl-long.453.pdfaclanthology.org
• Byte Latent Transformer: Patches Scale Better Than Tokens | Research - AI at Metaai.meta.com
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• How LLMs Actually Work | 0xkato0xkato.xyz
• You Can Learn Tokenization End-to-End with Reinforcement Learningarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• Shaping capabilities with token-level data filteringarxiv.org
• [2112.10508] Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLParxiv.org