The End of Tokenization - Ronald Yu's Substack

Some thoughts on tokenization and Meta's “Byte Latent Transformer: Patches Scale Better Than Tokens” paper

Meta’s new paper “Byte Latent Transformer: Patches Scale Better Than Tokens” eliminates LLMs’ dirtiest secret: tokenization. Instead of using a tokenizer, the Byte Latent Transformer, as the name implies, operates on the bytes of the raw text data. Previous works on eliminating tokenization have largely framed the central challenge as mitigating the fact that naively operating on raw bytes results in longer sequences and therefore extra compute for the same task. However, this is a losing battle as wasted compute will overshadow any minor benefits. The Byte Latent Transformer flips the…

saved by

• Gene Yang

related reading

• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Byte Latent Transformer: Patches Scale Better Than Tokensarxiv.org
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• 2025.acl-long.453.pdfaclanthology.org
• Byte Latent Transformer: Patches Scale Better Than Tokens | Research - AI at Metaai.meta.com
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• MambaByte: Token-free Selective State Space Modelarxiv.org
• How LLMs Actually Work | 0xkato0xkato.xyz
• LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
• You Can Learn Tokenization End-to-End with Reinforcement Learningarxiv.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• H-Nets - the Past | Goomba Labgoombalab.github.io