Language Modeling Without Neural Networks

Generating Shakespeare has become the “Hello World” of language models.1 Recently, I’ve been messing with alternative language models and came across unbounded …

Generating Shakespeare has become the "Hello World" of language models. 1 Recently, I've been messing with alternative language models and came across unbounded n-gram models. These models are purely statistical and don't require optimizing weights or training. A year ago, I read the paper Infini-gram , which scaled an unbounded n-gram model to trillions of tokens. While their model had applications helping guide neural LLMs during generation, standalone language generation was not explored. In this post, I'll explain how unbounded n-gram models work and how I improved their language generatio

saved by

• Devin Plumb

related reading

• Can gzip be a language model?nathan.rs
• Home | infini-graminfini-gram.io
• Language Modelinglena-voita.github.io
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• Introduction to Large Language Models | Machine Learning | Google for Developersdevelopers.google.com
• Recursive Language Models | Alex L. Zhangalexzhang13.github.io
• GenAI Handbookgenai-handbook.github.io
• A History of Large Language Modelsgregorygundersen.com
• Large Language Diffusion Modelsarxiv.org
• Large Language Model: world models or surface statistics?thegradient.pub
• Productizing Large Language Modelsblog.replit.com
• Alex L. Zhangalexzhang13.github.io